[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:28 dist-test-slave-45dx krb5kdc[465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:28 dist-test-slave-45dx krb5kdc[465](info): set up 2 sockets
Aug 06 11:41:28 dist-test-slave-45dx krb5kdc[465](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 11:41:30.097632 452 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.009s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:30 dist-test-slave-45dx krb5kdc[465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:30.222239 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:32855
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:32855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:30.354038 483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:30.354311 483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:30.354409 483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:30.358426 483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:30.358481 483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:30.358501 483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:30.358565 483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:30.358599 483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:30.364207 483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:32855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:32855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:30.365628 483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:30.366909 483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:30.379556 493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:30.379580 491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:30.380029 483 server_base.cc:1034] running on GCE node
W20260806 11:41:30.380643 497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:30.381340 483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:30.382939 483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:30.384109 483 hybrid_clock.cc:648] HybridClock initialized: now 1786016490384088 us; error 33 us; skew 500 ppm
Aug 06 11:41:30 dist-test-slave-45dx krb5kdc[465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:30.387151 483 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:30.388279 483 webserver.cc:460] Webserver started at http://127.0.113.62:40045/ using document root <none> and password file <none>
I20260806 11:41:30.389227 483 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:30.389330 483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:30.389731 483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:30.392279 483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "2cb2b00234dd420c90ee8ce64bfdbf70"
format_stamp: "Formatted at 2026-08-06 11:41:30 on dist-test-slave-45dx"
server_key: "bca11615079df623dbcb7ca15832cc8d"
server_key_iv: "a41f188c85de73581463e443eb318584"
server_key_version: "encryptionkey@0"
I20260806 11:41:30.392946 483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "2cb2b00234dd420c90ee8ce64bfdbf70"
format_stamp: "Formatted at 2026-08-06 11:41:30 on dist-test-slave-45dx"
server_key: "bca11615079df623dbcb7ca15832cc8d"
server_key_iv: "a41f188c85de73581463e443eb318584"
server_key_version: "encryptionkey@0"
I20260806 11:41:30.397806 483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 11:41:30.401262 505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:30.402854 483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 11:41:30.403072 483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "2cb2b00234dd420c90ee8ce64bfdbf70"
format_stamp: "Formatted at 2026-08-06 11:41:30 on dist-test-slave-45dx"
server_key: "bca11615079df623dbcb7ca15832cc8d"
server_key_iv: "a41f188c85de73581463e443eb318584"
server_key_version: "encryptionkey@0"
I20260806 11:41:30.403199 483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:30.415692 483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:30.418987 483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:30.419262 483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:30.435379 483 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:32855
I20260806 11:41:30.435580 579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:32855 every 8 connection(s)
I20260806 11:41:30.437129 483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:30.439148 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 483
I20260806 11:41:30.439283 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:30.440907 452 external_mini_cluster.cc:1428] Setting key 968b3c3f2db7dc09f1e1568b7218e6a7
I20260806 11:41:30.442478 580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:30.453305 580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70: Bootstrap starting.
I20260806 11:41:30.457788 580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:30.459470 580 log.cc:824] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:30.462560 580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70: No bootstrap required, opened a new log
Aug 06 11:41:30 dist-test-slave-45dx krb5kdc[465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:30.468204 580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb2b00234dd420c90ee8ce64bfdbf70" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 32855 } }
I20260806 11:41:30.468570 580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:30.468699 580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb2b00234dd420c90ee8ce64bfdbf70, State: Initialized, Role: FOLLOWER
I20260806 11:41:30.469480 580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb2b00234dd420c90ee8ce64bfdbf70" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 32855 } }
I20260806 11:41:30.469702 580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:30.469854 580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:30.469988 580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:30.471149 580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb2b00234dd420c90ee8ce64bfdbf70" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 32855 } }
I20260806 11:41:30.471851 580 leader_election.cc:302] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cb2b00234dd420c90ee8ce64bfdbf70; no voters:
I20260806 11:41:30.472415 580 leader_election.cc:288] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:30.472532 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:30.445874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.0.113.62:32855)
0806 11:41:30.446470 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:30.446538 (+ 68us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:30.447828 (+ 1290us) server_negotiation.cc:407] Connection header received
0806 11:41:30.452382 (+ 4554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:30.452401 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:30.452772 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:30.453102 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:30.455877 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.457257 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.458649 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.459145 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.459321 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:30.465038 (+ 5717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:30.465079 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:30.465093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:30.465123 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:30.468130 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:30.468909 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:30.468915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:30.468927 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:30.469039 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:30.469404 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:30.469408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:30.469409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:30.470044 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:30.470346 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:30.470977 (+ 631us) server_negotiation.cc:299] Negotiation successful
0806 11:41:30.471254 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":211,"threads_started":1}
I20260806 11:41:30.472647 592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:30.473035 592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb2b00234dd420c90ee8ce64bfdbf70, State: Running, Role: LEADER
I20260806 11:41:30.473495 592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb2b00234dd420c90ee8ce64bfdbf70" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 32855 } }
I20260806 11:41:30.474412 580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:30.477190 596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cb2b00234dd420c90ee8ce64bfdbf70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb2b00234dd420c90ee8ce64bfdbf70" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 32855 } } }
I20260806 11:41:30.477334 596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:30.477726 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cb2b00234dd420c90ee8ce64bfdbf70. Latest consensus state: current_term: 1 leader_uuid: "2cb2b00234dd420c90ee8ce64bfdbf70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb2b00234dd420c90ee8ce64bfdbf70" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 32855 } } }
I20260806 11:41:30.477840 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:30.478243 608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:30.483273 608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:30.491995 608 catalog_manager.cc:1347] Generated new cluster ID: 5faddeaabdce468896d72ea7d428f965
I20260806 11:41:30.492080 608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:30.516822 608 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:30.518244 608 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:30.531677 608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cb2b00234dd420c90ee8ce64bfdbf70: Generated new TSK 0
I20260806 11:41:30.532569 608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:41:30.567612 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:32855
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:30.694460 621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:30.694706 621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:30.694783 621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:30.698926 621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:30.698992 621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:30.699088 621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:30.704492 621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:32855
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:30.705572 621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:30.706691 621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 11:41:30.717378 621 server_base.cc:1034] running on GCE node
W20260806 11:41:30.717343 627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:30.717545 628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:30.717624 631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:30.718057 621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:30.718696 621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:30.719913 621 hybrid_clock.cc:648] HybridClock initialized: now 1786016490719886 us; error 45 us; skew 500 ppm
Aug 06 11:41:30 dist-test-slave-45dx krb5kdc[465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:30.723335 621 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:41:30.724234 621 webserver.cc:460] Webserver started at http://127.0.113.1:43279/ using document root <none> and password file <none>
I20260806 11:41:30.724821 621 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:30.724931 621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:30.725169 621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:30.727182 621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "e2c508dc840d4ae6b229b503f8cc5ed0"
format_stamp: "Formatted at 2026-08-06 11:41:30 on dist-test-slave-45dx"
server_key: "e67d90046c4e3f106ecfc2595800856c"
server_key_iv: "2ba1b348f943a03492e1b568ce0a974b"
server_key_version: "encryptionkey@0"
I20260806 11:41:30.727757 621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "e2c508dc840d4ae6b229b503f8cc5ed0"
format_stamp: "Formatted at 2026-08-06 11:41:30 on dist-test-slave-45dx"
server_key: "e67d90046c4e3f106ecfc2595800856c"
server_key_iv: "2ba1b348f943a03492e1b568ce0a974b"
server_key_version: "encryptionkey@0"
I20260806 11:41:30.731606 621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 11:41:30.734216 641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:30.735483 621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:41:30.735625 621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "e2c508dc840d4ae6b229b503f8cc5ed0"
format_stamp: "Formatted at 2026-08-06 11:41:30 on dist-test-slave-45dx"
server_key: "e67d90046c4e3f106ecfc2595800856c"
server_key_iv: "2ba1b348f943a03492e1b568ce0a974b"
server_key_version: "encryptionkey@0"
I20260806 11:41:30.735741 621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:30.749752 621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:30.752678 621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:30.752877 621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:30.753572 621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:30.754613 621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:30.754688 621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:30.754760 621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:30.754801 621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:30.765117 621 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:34953
I20260806 11:41:30.765148 798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:34953 every 8 connection(s)
I20260806 11:41:30.765997 621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:30.775314 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 621
I20260806 11:41:30.775457 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:30.775770 452 external_mini_cluster.cc:1428] Setting key cc57ba2e4664153a44e5e873722aaf46
Aug 06 11:41:30 dist-test-slave-45dx krb5kdc[465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:30.781491 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:30.768246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:44599 (local address 127.0.113.62:32855)
0806 11:41:30.768410 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:30.768413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:30.769230 (+ 817us) server_negotiation.cc:407] Connection header received
0806 11:41:30.771084 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:30.771088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:30.771152 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:30.771263 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:30.772490 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.773036 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.774186 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.774423 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.774532 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:30.777727 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:30.777730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:30.777733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:30.777761 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:30.779352 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:30.779884 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:30.779888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:30.779890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:30.779988 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:30.780357 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:30.780360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:30.780361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:30.780562 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:30.780676 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:30.781186 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 11:41:30.781315 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:41:30.782318 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:30.768621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:32855 (local address 127.0.113.1:44599)
0806 11:41:30.769072 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:30.769115 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:30.770739 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:30.771454 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:30.771460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:30.771815 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:30.772278 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:30.772290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.773169 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.773172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:30.774053 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:30.774062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.774585 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.774589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:30.774755 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:30.775428 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:30.775450 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:30.777536 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:30.779483 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:30.779489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:30.779503 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:30.779770 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:30.780083 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:30.780086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:30.780088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:30.780232 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:30.780675 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:30.780683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:30.780940 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 11:41:30.781127 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 11:41:30.781445 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260806 11:41:30.783535 799 heartbeater.cc:347] Connected to a master server at 127.0.113.62:32855
I20260806 11:41:30.783746 799 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:30.784176 799 heartbeater.cc:511] Master 127.0.113.62:32855 requested a full tablet report, sending...
I20260806 11:41:30.786518 532 ts_manager.cc:194] Registered new tserver with Master: e2c508dc840d4ae6b229b503f8cc5ed0 (127.0.113.1:34953)
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:41:30.788427 532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:44599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:41:30.805620 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:32855
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:30.917325 805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:30.917558 805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:30.917598 805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:30.921470 805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:30.921526 805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:30.921581 805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:41:30.926656 805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:32855
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:30.927582 805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:30.928495 805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:30.937918 811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:30.937963 805 server_base.cc:1034] running on GCE node
W20260806 11:41:30.937943 812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:30.937918 814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:30.938584 805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:30.939159 805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:30.940356 805 hybrid_clock.cc:648] HybridClock initialized: now 1786016490940317 us; error 42 us; skew 500 ppm
Aug 06 11:41:30 dist-test-slave-45dx krb5kdc[465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:30.943073 805 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:41:30.943928 805 webserver.cc:460] Webserver started at http://127.0.113.2:42867/ using document root <none> and password file <none>
I20260806 11:41:30.944566 805 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:30.944661 805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:30.944900 805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:30.946911 805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "048d338934fb451fb565994b8bdf53be"
format_stamp: "Formatted at 2026-08-06 11:41:30 on dist-test-slave-45dx"
server_key: "a2a2ef11ef4be20b0c2b1306dd9e5fa5"
server_key_iv: "b7f392c52260e98d342c43dea9d6cd15"
server_key_version: "encryptionkey@0"
I20260806 11:41:30.947491 805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "048d338934fb451fb565994b8bdf53be"
format_stamp: "Formatted at 2026-08-06 11:41:30 on dist-test-slave-45dx"
server_key: "a2a2ef11ef4be20b0c2b1306dd9e5fa5"
server_key_iv: "b7f392c52260e98d342c43dea9d6cd15"
server_key_version: "encryptionkey@0"
I20260806 11:41:30.951220 805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260806 11:41:30.953739 824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:30.954982 805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:41:30.955121 805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "048d338934fb451fb565994b8bdf53be"
format_stamp: "Formatted at 2026-08-06 11:41:30 on dist-test-slave-45dx"
server_key: "a2a2ef11ef4be20b0c2b1306dd9e5fa5"
server_key_iv: "b7f392c52260e98d342c43dea9d6cd15"
server_key_version: "encryptionkey@0"
I20260806 11:41:30.955241 805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:30.971501 805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:30.974859 805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:30.975076 805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:30.975698 805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:30.976600 805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:30.976670 805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:30.976739 805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:30.976783 805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:30.986215 805 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:40227
I20260806 11:41:30.986240 978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:40227 every 8 connection(s)
I20260806 11:41:30.987082 805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:41:30.992573 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 805
I20260806 11:41:30.992697 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:41:30.992964 452 external_mini_cluster.cc:1428] Setting key 8888c53bc561c8212601392cf7b4758f
Aug 06 11:41:30 dist-test-slave-45dx krb5kdc[465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:31.002310 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:30.988931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:46533 (local address 127.0.113.62:32855)
0806 11:41:30.989075 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:30.989080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:30.989940 (+ 860us) server_negotiation.cc:407] Connection header received
0806 11:41:30.991198 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:30.991201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:30.991246 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:30.991319 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:30.992580 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.993343 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.994962 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.995327 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.995462 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:30.998010 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:30.998016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:30.998022 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:30.998054 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:30.999767 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:31.000464 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:31.000472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:31.000475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:31.000579 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:31.000949 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:31.000956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:31.000960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:31.001225 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:31.001363 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:31.001907 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 11:41:31.002061 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:41:31.003016 981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:30.989331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:32855 (local address 127.0.113.2:46533)
0806 11:41:30.989792 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:30.989833 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:30.991010 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:30.991490 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:30.991496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:30.991848 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:30.992419 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:30.992436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.993495 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.993499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:30.994739 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:30.994749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:30.995487 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:30.995490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:30.995630 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:30.996288 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:30.996342 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:30.997823 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:30.999918 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:30.999924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:30.999935 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:31.000255 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:31.000681 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:31.000684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:31.000685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:31.000836 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:31.001358 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:31.001363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:31.001620 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 11:41:31.001795 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 11:41:31.002089 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260806 11:41:31.004379 979 heartbeater.cc:347] Connected to a master server at 127.0.113.62:32855
I20260806 11:41:31.004653 979 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:41:31.005277 979 heartbeater.cc:511] Master 127.0.113.62:32855 requested a full tablet report, sending...
I20260806 11:41:31.006494 532 ts_manager.cc:194] Registered new tserver with Master: 048d338934fb451fb565994b8bdf53be (127.0.113.2:40227)
I20260806 11:41:31.007081 532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:46533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:41:31.026753 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:32855
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:31.141670 984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:31.141952 984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:31.141997 984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:31.145908 984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:31.145963 984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:31.146020 984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:41:31.150946 984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:32855
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:31.151850 984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:31.152940 984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:31.162441 995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:31.162456 992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:31.162725 984 server_base.cc:1034] running on GCE node
W20260806 11:41:31.162482 990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:31.163296 984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:31.163946 984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:31.165143 984 hybrid_clock.cc:648] HybridClock initialized: now 1786016491165123 us; error 31 us; skew 500 ppm
Aug 06 11:41:31 dist-test-slave-45dx krb5kdc[465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:31.168016 984 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:41:31.168949 984 webserver.cc:460] Webserver started at http://127.0.113.3:44611/ using document root <none> and password file <none>
I20260806 11:41:31.169559 984 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:31.169654 984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:31.169904 984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:31.171973 984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "f82d12ce6f784add894e35b0bafe07bc"
format_stamp: "Formatted at 2026-08-06 11:41:31 on dist-test-slave-45dx"
server_key: "157684dbd99feb83fa1059358a054f8c"
server_key_iv: "cbcf92da7ac32b6e191a04bff894e0ad"
server_key_version: "encryptionkey@0"
I20260806 11:41:31.172571 984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "f82d12ce6f784add894e35b0bafe07bc"
format_stamp: "Formatted at 2026-08-06 11:41:31 on dist-test-slave-45dx"
server_key: "157684dbd99feb83fa1059358a054f8c"
server_key_iv: "cbcf92da7ac32b6e191a04bff894e0ad"
server_key_version: "encryptionkey@0"
I20260806 11:41:31.176505 984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 11:41:31.178987 1003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:31.180150 984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:31.180356 984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "f82d12ce6f784add894e35b0bafe07bc"
format_stamp: "Formatted at 2026-08-06 11:41:31 on dist-test-slave-45dx"
server_key: "157684dbd99feb83fa1059358a054f8c"
server_key_iv: "cbcf92da7ac32b6e191a04bff894e0ad"
server_key_version: "encryptionkey@0"
I20260806 11:41:31.180495 984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:31.201203 984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:31.204103 984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:31.204322 984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:31.204919 984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:31.205744 984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:31.205817 984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:31.205889 984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:31.205943 984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:31.215644 984 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:39325
I20260806 11:41:31.215673 1169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:39325 every 8 connection(s)
I20260806 11:41:31.216619 984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:41:31.224021 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 984
I20260806 11:41:31.224161 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:41:31.224445 452 external_mini_cluster.cc:1428] Setting key 3f5caef1f3b5c1a9d03a731fa02f65a6
Aug 06 11:41:31 dist-test-slave-45dx krb5kdc[465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:31.230535 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:31.218474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:60421 (local address 127.0.113.62:32855)
0806 11:41:31.218611 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:31.218614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:31.219445 (+ 831us) server_negotiation.cc:407] Connection header received
0806 11:41:31.220840 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:31.220843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:31.220892 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:31.220971 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:31.222085 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:31.222603 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:31.223682 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:31.224005 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:31.224125 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:31.226984 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:31.226987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:31.226989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:31.227013 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:31.228495 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:31.229019 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:31.229022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:31.229024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:31.229114 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:31.229465 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:31.229469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:31.229471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:31.229652 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:31.229762 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:31.230249 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 11:41:31.230361 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 11:41:31.231267 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:31.218844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:32855 (local address 127.0.113.3:60421)
0806 11:41:31.219282 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:31.219324 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:31.220641 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:31.221135 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:31.221142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:31.221479 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:31.221954 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:31.221967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:31.222709 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:31.222712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:31.223571 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:31.223581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:31.224201 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:31.224207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:31.224440 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:31.225129 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:31.225155 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:31.226786 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:31.228618 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:31.228624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:31.228637 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:31.228910 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:31.229211 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:31.229215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:31.229217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:31.229361 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:31.229762 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:31.229770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:31.230022 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 11:41:31.230208 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 11:41:31.230509 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260806 11:41:31.232366 1170 heartbeater.cc:347] Connected to a master server at 127.0.113.62:32855
I20260806 11:41:31.232622 1170 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:31.233117 1170 heartbeater.cc:511] Master 127.0.113.62:32855 requested a full tablet report, sending...
I20260806 11:41:31.234140 532 ts_manager.cc:194] Registered new tserver with Master: f82d12ce6f784add894e35b0bafe07bc (127.0.113.3:39325)
I20260806 11:41:31.234695 532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:60421
I20260806 11:41:31.239107 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 11:41:31 dist-test-slave-45dx krb5kdc[465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 11:41:31 dist-test-slave-45dx krb5kdc[465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:41:31.267096 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:31.257567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.0.113.1:34953)
0806 11:41:31.257881 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:31.257900 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:31.257922 (+ 22us) server_negotiation.cc:407] Connection header received
0806 11:41:31.257972 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:31.257976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:31.258152 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:31.258336 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:31.259072 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:31.259919 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:31.260806 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:31.261245 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:31.261366 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:31.263763 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:31.263771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:31.263783 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:31.263804 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:31.265506 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:31.265943 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:31.265948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:31.265949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:31.266029 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:31.266274 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:31.266276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:31.266277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:31.266496 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:31.266668 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:31.266825 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 11:41:31.266949 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
W20260806 11:41:31.268205 679 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46184
Aug 06 11:41:31 dist-test-slave-45dx krb5kdc[465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 11:41:31 dist-test-slave-45dx krb5kdc[465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:41:31.294948 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:31.287047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.0.113.1:34953)
0806 11:41:31.287211 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:31.287214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:31.287269 (+ 55us) server_negotiation.cc:407] Connection header received
0806 11:41:31.287387 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:31.287389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:31.287429 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:31.287497 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:31.288125 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:31.288684 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:31.289511 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:31.289727 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:31.289800 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:31.292117 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:31.292120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:31.292122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:31.292161 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:31.293447 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:31.293840 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:31.293843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:31.293844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:31.293921 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:31.294154 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:31.294156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:31.294157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:31.294307 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:31.294415 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:31.294636 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 11:41:31.294729 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 11:41:31.296137 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 621
I20260806 11:41:31.308566 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 805
I20260806 11:41:31.318367 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 984
I20260806 11:41:31.336601 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 483
2026-08-06T11:41:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2502 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:31 dist-test-slave-45dx krb5kdc[1195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:31 dist-test-slave-45dx krb5kdc[1195](info): set up 2 sockets
Aug 06 11:41:31 dist-test-slave-45dx krb5kdc[1195](info): commencing operation
krb5kdc: starting...
W20260806 11:41:32.563437 452 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:32 dist-test-slave-45dx krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:32.693128 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:41193
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:34075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:41193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:32.803395 1213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:32.803663 1213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:32.803704 1213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:32.807662 1213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:32.807715 1213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:32.807734 1213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:32.807750 1213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:32.807765 1213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:32.812906 1213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:41193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:41193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:32.813930 1213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:32.814946 1213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:32.823793 1224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:32.823793 1219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:32.823971 1213 server_base.cc:1034] running on GCE node
W20260806 11:41:32.823774 1220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:32.824827 1213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:32.825927 1213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:32.827142 1213 hybrid_clock.cc:648] HybridClock initialized: now 1786016492827120 us; error 31 us; skew 500 ppm
Aug 06 11:41:32 dist-test-slave-45dx krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:32.836925 1213 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:32.837792 1213 webserver.cc:460] Webserver started at http://127.0.113.62:38053/ using document root <none> and password file <none>
I20260806 11:41:32.838349 1213 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:32.838445 1213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:32.838675 1213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:32.840721 1213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "7e39855deecf431ea2bb5a3e491b3507"
format_stamp: "Formatted at 2026-08-06 11:41:32 on dist-test-slave-45dx"
server_key: "edd2fce2b3beb3facf26483b392db22b"
server_key_iv: "f76a0fc73af161a4988048aa263006a0"
server_key_version: "encryptionkey@0"
I20260806 11:41:32.841336 1213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "7e39855deecf431ea2bb5a3e491b3507"
format_stamp: "Formatted at 2026-08-06 11:41:32 on dist-test-slave-45dx"
server_key: "edd2fce2b3beb3facf26483b392db22b"
server_key_iv: "f76a0fc73af161a4988048aa263006a0"
server_key_version: "encryptionkey@0"
I20260806 11:41:32.845185 1213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 11:41:32.847703 1237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:32.849032 1213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:32.849190 1213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "7e39855deecf431ea2bb5a3e491b3507"
format_stamp: "Formatted at 2026-08-06 11:41:32 on dist-test-slave-45dx"
server_key: "edd2fce2b3beb3facf26483b392db22b"
server_key_iv: "f76a0fc73af161a4988048aa263006a0"
server_key_version: "encryptionkey@0"
I20260806 11:41:32.849313 1213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:32.868685 1213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:32.871776 1213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:32.872025 1213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:32.879807 1213 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:41193
I20260806 11:41:32.879828 1307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:41193 every 8 connection(s)
I20260806 11:41:32.880846 1213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:32.883697 1308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:32.889209 1308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507: Bootstrap starting.
I20260806 11:41:32.889547 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 1213
I20260806 11:41:32.889683 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:32.889940 452 external_mini_cluster.cc:1428] Setting key c7f8d6c8999499d0e50c621113079801
I20260806 11:41:32.891810 1308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:32.892606 1308 log.cc:824] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:32.894402 1308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507: No bootstrap required, opened a new log
I20260806 11:41:32.898017 1308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e39855deecf431ea2bb5a3e491b3507" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41193 } }
I20260806 11:41:32.898279 1308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:32.898362 1308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e39855deecf431ea2bb5a3e491b3507, State: Initialized, Role: FOLLOWER
Aug 06 11:41:32 dist-test-slave-45dx krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:32.898880 1308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e39855deecf431ea2bb5a3e491b3507" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41193 } }
I20260806 11:41:32.899051 1308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:32.899133 1308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:32.899240 1308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:32.900457 1308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e39855deecf431ea2bb5a3e491b3507" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41193 } }
I20260806 11:41:32.900858 1308 leader_election.cc:302] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e39855deecf431ea2bb5a3e491b3507; no voters:
I20260806 11:41:32.901261 1308 leader_election.cc:288] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:32.901345 1314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:32.901572 1314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [term 1 LEADER]: Becoming Leader. State: Replica: 7e39855deecf431ea2bb5a3e491b3507, State: Running, Role: LEADER
I20260806 11:41:32.901952 1314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e39855deecf431ea2bb5a3e491b3507" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41193 } }
I20260806 11:41:32.902647 1308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:32.903743 1318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e39855deecf431ea2bb5a3e491b3507. Latest consensus state: current_term: 1 leader_uuid: "7e39855deecf431ea2bb5a3e491b3507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e39855deecf431ea2bb5a3e491b3507" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41193 } } }
I20260806 11:41:32.903867 1318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:32.904204 1317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e39855deecf431ea2bb5a3e491b3507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e39855deecf431ea2bb5a3e491b3507" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41193 } } }
I20260806 11:41:32.904345 1317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:32.905141 1326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:32.905673 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:32.891304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.0.113.62:41193)
0806 11:41:32.891757 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:32.891772 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:32.891807 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:41:32.892537 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:32.892555 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:32.892924 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:32.893251 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:32.894051 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:32.895072 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:32.895954 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:32.896511 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:32.896688 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:32.899507 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:32.899512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:32.899526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:32.899556 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:32.901656 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:32.902207 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:32.902212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:32.902219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:32.902370 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:32.902676 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:32.902680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:32.902682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:32.903234 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:32.904176 (+ 942us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:32.904367 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 11:41:32.904678 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":135,"threads_started":1}
I20260806 11:41:32.908026 1326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:32.913667 1326 catalog_manager.cc:1347] Generated new cluster ID: cbd45dd3e621471c9122925a3a520324
I20260806 11:41:32.913733 1326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:32.932296 1326 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:32.933632 1326 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:32.942160 1326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e39855deecf431ea2bb5a3e491b3507: Generated new TSK 0
I20260806 11:41:32.942955 1326 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:41:32.980255 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41193
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:34075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:33.099908 1343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:33.100178 1343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:33.100261 1343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:33.104379 1343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:33.104451 1343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:33.104537 1343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:33.109755 1343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41193
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:33.110834 1343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:33.111881 1343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:33.121486 1356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:33.121457 1352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:33.121445 1353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:33.121917 1343 server_base.cc:1034] running on GCE node
I20260806 11:41:33.122309 1343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:33.122928 1343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:33.124131 1343 hybrid_clock.cc:648] HybridClock initialized: now 1786016493124106 us; error 40 us; skew 500 ppm
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:33.127038 1343 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:41:33.127857 1343 webserver.cc:460] Webserver started at http://127.0.113.1:34063/ using document root <none> and password file <none>
I20260806 11:41:33.128466 1343 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:33.128562 1343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:33.128790 1343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:33.130743 1343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "bbb6ada01dd14acaa3057cc831ae557d"
format_stamp: "Formatted at 2026-08-06 11:41:33 on dist-test-slave-45dx"
server_key: "42e95ca39107b83958a4012b2fc8647c"
server_key_iv: "ab88128b43099becbb16750ca9c5879a"
server_key_version: "encryptionkey@0"
I20260806 11:41:33.131295 1343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "bbb6ada01dd14acaa3057cc831ae557d"
format_stamp: "Formatted at 2026-08-06 11:41:33 on dist-test-slave-45dx"
server_key: "42e95ca39107b83958a4012b2fc8647c"
server_key_iv: "ab88128b43099becbb16750ca9c5879a"
server_key_version: "encryptionkey@0"
I20260806 11:41:33.134899 1343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:41:33.137300 1365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:33.138449 1343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:33.138579 1343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "bbb6ada01dd14acaa3057cc831ae557d"
format_stamp: "Formatted at 2026-08-06 11:41:33 on dist-test-slave-45dx"
server_key: "42e95ca39107b83958a4012b2fc8647c"
server_key_iv: "ab88128b43099becbb16750ca9c5879a"
server_key_version: "encryptionkey@0"
I20260806 11:41:33.138686 1343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:33.170257 1343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:33.173847 1343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:33.174079 1343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:33.174707 1343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:33.175558 1343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:33.175629 1343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:33.175704 1343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:33.175757 1343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:33.185297 1343 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:38465
I20260806 11:41:33.185325 1534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:38465 every 8 connection(s)
I20260806 11:41:33.186178 1343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:33.187942 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 1343
I20260806 11:41:33.188083 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:33.188409 452 external_mini_cluster.cc:1428] Setting key 68c37689bb2d9213728e2b0105e24e56
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:41:33.202057 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.188129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:59365 (local address 127.0.113.62:41193)
0806 11:41:33.188280 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.188285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.189338 (+ 1053us) server_negotiation.cc:407] Connection header received
0806 11:41:33.191135 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.191138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.191182 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.191249 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:33.192791 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.193628 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.194771 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.195223 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.195344 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.197970 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:33.197974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:33.197977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:33.198008 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:33.199799 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.200395 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.200398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.200399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.200506 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.200901 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.200904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.200906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.201095 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:33.201206 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.201719 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.201879 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:41:33.203111 1540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.188616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:41193 (local address 127.0.113.1:59365)
0806 11:41:33.189161 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:33.189216 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:33.190894 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:33.191441 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:33.191452 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:33.191975 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:33.192632 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.192646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.193764 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.193767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.194638 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.194647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.195377 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.195380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.195523 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.196218 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:33.196234 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:33.197785 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:33.199970 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:33.199976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:33.199986 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:33.200243 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:33.200613 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:33.200617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:33.200619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:33.200774 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:33.201208 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:33.201217 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:33.201474 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 11:41:33.201660 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 11:41:33.202006 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":185,"threads_started":1}
I20260806 11:41:33.205643 1535 heartbeater.cc:347] Connected to a master server at 127.0.113.62:41193
I20260806 11:41:33.205869 1535 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:33.206337 1535 heartbeater.cc:511] Master 127.0.113.62:41193 requested a full tablet report, sending...
I20260806 11:41:33.207839 1259 ts_manager.cc:194] Registered new tserver with Master: bbb6ada01dd14acaa3057cc831ae557d (127.0.113.1:38465)
I20260806 11:41:33.209038 1259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:59365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:41:33.220671 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41193
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:34075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:33.333556 1543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:33.333817 1543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:33.333861 1543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:33.337903 1543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:33.337966 1543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:33.338024 1543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:41:33.343326 1543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41193
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:33.344287 1543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:33.345348 1543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:33.355036 1553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:33.355049 1550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:33.355041 1551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:33.355396 1543 server_base.cc:1034] running on GCE node
I20260806 11:41:33.355808 1543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:33.356511 1543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:33.357683 1543 hybrid_clock.cc:648] HybridClock initialized: now 1786016493357675 us; error 31 us; skew 500 ppm
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:33.360499 1543 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:41:33.361471 1543 webserver.cc:460] Webserver started at http://127.0.113.2:38459/ using document root <none> and password file <none>
I20260806 11:41:33.362053 1543 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:33.362152 1543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:33.362387 1543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:33.364518 1543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "1c0ccefb31fc4de6a5409090b66adf00"
format_stamp: "Formatted at 2026-08-06 11:41:33 on dist-test-slave-45dx"
server_key: "2ab62904365c6ed14fa1d0ca692f0ba4"
server_key_iv: "ebbc77b9398220398a419dfbed9abe11"
server_key_version: "encryptionkey@0"
I20260806 11:41:33.365139 1543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "1c0ccefb31fc4de6a5409090b66adf00"
format_stamp: "Formatted at 2026-08-06 11:41:33 on dist-test-slave-45dx"
server_key: "2ab62904365c6ed14fa1d0ca692f0ba4"
server_key_iv: "ebbc77b9398220398a419dfbed9abe11"
server_key_version: "encryptionkey@0"
I20260806 11:41:33.368952 1543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 11:41:33.371444 1563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:33.372782 1543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260806 11:41:33.372922 1543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "1c0ccefb31fc4de6a5409090b66adf00"
format_stamp: "Formatted at 2026-08-06 11:41:33 on dist-test-slave-45dx"
server_key: "2ab62904365c6ed14fa1d0ca692f0ba4"
server_key_iv: "ebbc77b9398220398a419dfbed9abe11"
server_key_version: "encryptionkey@0"
I20260806 11:41:33.373047 1543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:33.388439 1543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:33.391719 1543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:33.391928 1543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:33.392570 1543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:33.393411 1543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:33.393484 1543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:33.393558 1543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:33.393603 1543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:33.402520 1543 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:42925
I20260806 11:41:33.402547 1714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:42925 every 8 connection(s)
I20260806 11:41:33.403504 1543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:41:33.407771 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 1543
I20260806 11:41:33.407917 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:41:33.408178 452 external_mini_cluster.cc:1428] Setting key 009c032e1c7644fb658bfae04305218e
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:33.417721 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.405439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:47115 (local address 127.0.113.62:41193)
0806 11:41:33.405593 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.405596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.406451 (+ 855us) server_negotiation.cc:407] Connection header received
0806 11:41:33.407815 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.407818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.407876 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.408001 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:33.409232 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.409732 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.410866 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.411086 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.411198 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.413872 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:33.413876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:33.413878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:33.413902 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:33.415373 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.416020 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.416023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.416024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.416098 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.416557 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.416560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.416561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.416730 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:33.416809 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.417447 (+ 638us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.417547 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 11:41:33.419028 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.405834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:41193 (local address 127.0.113.2:47115)
0806 11:41:33.406293 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:33.406334 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:33.407600 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:33.408178 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:33.408184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:33.408635 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:33.409078 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.409092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.409860 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.409863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.410729 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.410739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.411236 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.411240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.411402 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.411990 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:33.412007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:33.413677 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:33.415500 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:33.415509 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:33.415526 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:33.415917 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:33.416197 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:33.416203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:33.416206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:33.416441 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:33.416845 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:33.416854 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:33.417179 (+ 325us) client_negotiation.cc:769] Sending connection context
0806 11:41:33.417398 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 11:41:33.417780 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:41:33.420253 1715 heartbeater.cc:347] Connected to a master server at 127.0.113.62:41193
I20260806 11:41:33.420506 1715 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:33.421048 1715 heartbeater.cc:511] Master 127.0.113.62:41193 requested a full tablet report, sending...
I20260806 11:41:33.422103 1259 ts_manager.cc:194] Registered new tserver with Master: 1c0ccefb31fc4de6a5409090b66adf00 (127.0.113.2:42925)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
I20260806 11:41:33.422667 1259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:47115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:41:33.438712 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41193
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:34075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:33.549764 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:33.550061 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:33.550140 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:33.554268 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:33.554347 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:33.554438 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:41:33.559587 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41193
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:33.560643 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:33.561666 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:33.570930 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:33.570928 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:33.571022 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:33.571139 1720 server_base.cc:1034] running on GCE node
I20260806 11:41:33.571590 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:33.572242 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:33.573473 1720 hybrid_clock.cc:648] HybridClock initialized: now 1786016493573455 us; error 29 us; skew 500 ppm
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:33.576152 1720 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:41:33.577099 1720 webserver.cc:460] Webserver started at http://127.0.113.3:38115/ using document root <none> and password file <none>
I20260806 11:41:33.577639 1720 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:33.577729 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:33.577988 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:33.579957 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "e018388a5f6a4372ab79985aa31207ad"
format_stamp: "Formatted at 2026-08-06 11:41:33 on dist-test-slave-45dx"
server_key: "ca42015d1734650e36f74c3fe9f59515"
server_key_iv: "067803f3b73632926a7ba9a9404686d1"
server_key_version: "encryptionkey@0"
I20260806 11:41:33.580547 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "e018388a5f6a4372ab79985aa31207ad"
format_stamp: "Formatted at 2026-08-06 11:41:33 on dist-test-slave-45dx"
server_key: "ca42015d1734650e36f74c3fe9f59515"
server_key_iv: "067803f3b73632926a7ba9a9404686d1"
server_key_version: "encryptionkey@0"
I20260806 11:41:33.584036 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 11:41:33.586436 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:33.587508 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 11:41:33.587639 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "e018388a5f6a4372ab79985aa31207ad"
format_stamp: "Formatted at 2026-08-06 11:41:33 on dist-test-slave-45dx"
server_key: "ca42015d1734650e36f74c3fe9f59515"
server_key_iv: "067803f3b73632926a7ba9a9404686d1"
server_key_version: "encryptionkey@0"
I20260806 11:41:33.587747 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:33.596251 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:33.599000 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:33.599191 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:33.599728 1720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:33.600580 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:33.600651 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:33.600713 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:33.600764 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:33.610146 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:33799
I20260806 11:41:33.610183 1896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:33799 every 8 connection(s)
I20260806 11:41:33.610976 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:41:33.615329 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 1720
I20260806 11:41:33.615466 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:41:33.615767 452 external_mini_cluster.cc:1428] Setting key e0682b773d1e4f241cdd6615c3dfbf3f
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:33.625110 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.612777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:47151 (local address 127.0.113.62:41193)
0806 11:41:33.612917 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.612921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.613767 (+ 846us) server_negotiation.cc:407] Connection header received
0806 11:41:33.615164 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.615168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.615221 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.615296 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:33.616817 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.617451 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.618818 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.619055 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.619140 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.621690 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:33.621693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:33.621695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:33.621720 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:33.623006 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.623540 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.623545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.623547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.623635 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.623986 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.623990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.623992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.624178 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:33.624289 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.624802 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.624937 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 11:41:33.625877 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.613149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:41193 (local address 127.0.113.3:47151)
0806 11:41:33.613623 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:33.613664 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:33.614909 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:33.615525 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:33.615534 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:33.616006 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:33.616671 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.616688 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.617583 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.617586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.618706 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.618716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.619192 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.619195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.619342 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.619898 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:33.619915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:33.621521 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:33.623137 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:33.623143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:33.623157 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:33.623433 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:33.623732 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:33.623736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:33.623738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:33.623886 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:33.624288 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:33.624295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:33.624564 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 11:41:33.624757 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 11:41:33.625078 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":121,"threads_started":1}
I20260806 11:41:33.626950 1897 heartbeater.cc:347] Connected to a master server at 127.0.113.62:41193
I20260806 11:41:33.627195 1897 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:33.627679 1897 heartbeater.cc:511] Master 127.0.113.62:41193 requested a full tablet report, sending...
I20260806 11:41:33.628821 1259 ts_manager.cc:194] Registered new tserver with Master: e018388a5f6a4372ab79985aa31207ad (127.0.113.3:33799)
I20260806 11:41:33.629482 1259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:47151
I20260806 11:41:33.629921 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:41:33.641167 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.634007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.0.113.62:41193)
0806 11:41:33.634163 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.634167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.634314 (+ 147us) server_negotiation.cc:407] Connection header received
0806 11:41:33.634462 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.634465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.634515 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.634599 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:33.635304 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.635806 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.636691 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.636916 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.637023 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.638253 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:33.638257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:33.638259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:33.638284 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:33.639457 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.639868 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.639872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.639874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.639957 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.640243 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.640247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.640248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.640521 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:33.640657 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.640906 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.641016 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 11:41:33.646566 1259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:41:33.649827 1259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:33.665867 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.659279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:42925 (local address 127.0.0.1:54956)
0806 11:41:33.659810 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:33.659862 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:33.659988 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:33.661284 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:33.661291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:33.661315 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:33.661452 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.661460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.662956 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.662960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.664339 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.664350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.665312 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.665316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.665443 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.665483 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 11:41:33.665600 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 11:41:33.665716 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":83,"threads_started":1}
I20260806 11:41:33.665874 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.659487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.0.113.2:42925)
0806 11:41:33.660301 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.660343 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.660370 (+ 27us) server_negotiation.cc:407] Connection header received
0806 11:41:33.660459 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.660464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.661118 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.661296 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:33.661639 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.662794 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.664491 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.665187 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.665311 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.665505 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.665634 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.665721 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":165,"threads_started":1}
I20260806 11:41:33.666314 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.659706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:38465 (local address 127.0.0.1:49330)
0806 11:41:33.661069 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:33.661087 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:33.661271 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:33.662017 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:33.662021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:33.662039 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:33.662139 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.662144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.663561 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.663565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.664682 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.664692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.665896 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.665899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.665995 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.666029 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 11:41:33.666114 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 11:41:33.666184 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"thread_start_us":102,"threads_started":1}
I20260806 11:41:33.666896 1920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.659991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:33799 (local address 127.0.0.1:49524)
0806 11:41:33.661833 (+ 1842us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:33.661848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:33.662006 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:33.662420 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:33.662423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:33.662443 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:33.662528 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.662534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.663821 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.663825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.664627 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.664637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.665883 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.665887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.666568 (+ 681us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.666594 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:41:33.666700 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 11:41:33.666774 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":97,"threads_started":1}
I20260806 11:41:33.667044 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.660792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.0.113.3:33799)
0806 11:41:33.661139 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.661151 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.662008 (+ 857us) server_negotiation.cc:407] Connection header received
0806 11:41:33.662099 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.662107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.662274 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.662432 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:33.662681 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.663699 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.664749 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.665554 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.665671 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.665813 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.666778 (+ 965us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.666877 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":172,"threads_started":1}
I20260806 11:41:33.667140 1919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.660155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.0.113.1:38465)
0806 11:41:33.660574 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.660590 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.661184 (+ 594us) server_negotiation.cc:407] Connection header received
0806 11:41:33.661463 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.661470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.661657 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.661802 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:33.662251 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.663411 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.664851 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.665632 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.665722 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.665833 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.666151 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.666970 (+ 819us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":149,"threads_started":1}
I20260806 11:41:33.669255 1624 tablet_service.cc:1467] Processing CreateTablet for tablet c283753528fe4059980f36368ac8fe8b (DEFAULT_TABLE table=test-table [id=38fd44c7120f44459cec479262583806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:33.669508 1437 tablet_service.cc:1467] Processing CreateTablet for tablet c283753528fe4059980f36368ac8fe8b (DEFAULT_TABLE table=test-table [id=38fd44c7120f44459cec479262583806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:33.670580 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c283753528fe4059980f36368ac8fe8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:33.670799 1624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c283753528fe4059980f36368ac8fe8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:33.670840 1805 tablet_service.cc:1467] Processing CreateTablet for tablet c283753528fe4059980f36368ac8fe8b (DEFAULT_TABLE table=test-table [id=38fd44c7120f44459cec479262583806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:33.671746 1805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c283753528fe4059980f36368ac8fe8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:33.677367 1922 tablet_bootstrap.cc:492] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00: Bootstrap starting.
I20260806 11:41:33.678514 1923 tablet_bootstrap.cc:492] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d: Bootstrap starting.
I20260806 11:41:33.680017 1922 tablet_bootstrap.cc:654] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:33.680197 1923 tablet_bootstrap.cc:654] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:33.680478 1924 tablet_bootstrap.cc:492] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad: Bootstrap starting.
I20260806 11:41:33.681159 1922 log.cc:824] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:33.681569 1923 log.cc:824] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:33.682515 1924 tablet_bootstrap.cc:654] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:33.682991 1922 tablet_bootstrap.cc:492] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00: No bootstrap required, opened a new log
I20260806 11:41:33.683187 1922 ts_tablet_manager.cc:1397] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 11:41:33.683593 1924 log.cc:824] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:33.685226 1923 tablet_bootstrap.cc:492] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d: No bootstrap required, opened a new log
I20260806 11:41:33.685429 1923 ts_tablet_manager.cc:1397] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 11:41:33.685523 1924 tablet_bootstrap.cc:492] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad: No bootstrap required, opened a new log
I20260806 11:41:33.685708 1924 ts_tablet_manager.cc:1397] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:41:33.686790 1922 raft_consensus.cc:348] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } }
I20260806 11:41:33.687186 1922 raft_consensus.cc:374] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:33.687289 1922 raft_consensus.cc:729] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0ccefb31fc4de6a5409090b66adf00, State: Initialized, Role: FOLLOWER
I20260806 11:41:33.687826 1922 consensus_queue.cc:260] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } }
I20260806 11:41:33.688745 1715 heartbeater.cc:503] Master 127.0.113.62:41193 was elected leader, sending a full tablet report...
I20260806 11:41:33.688673 1923 raft_consensus.cc:348] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } }
I20260806 11:41:33.688920 1923 raft_consensus.cc:374] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:33.688977 1923 raft_consensus.cc:729] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb6ada01dd14acaa3057cc831ae557d, State: Initialized, Role: FOLLOWER
I20260806 11:41:33.689375 1923 consensus_queue.cc:260] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } }
I20260806 11:41:33.689548 1924 raft_consensus.cc:348] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } }
I20260806 11:41:33.689857 1924 raft_consensus.cc:374] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:33.689970 1924 raft_consensus.cc:729] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e018388a5f6a4372ab79985aa31207ad, State: Initialized, Role: FOLLOWER
I20260806 11:41:33.690097 1535 heartbeater.cc:503] Master 127.0.113.62:41193 was elected leader, sending a full tablet report...
I20260806 11:41:33.690310 1923 ts_tablet_manager.cc:1428] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 11:41:33.690428 1924 consensus_queue.cc:260] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } }
I20260806 11:41:33.690922 1922 ts_tablet_manager.cc:1428] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 11:41:33.691425 1924 ts_tablet_manager.cc:1428] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 11:41:33.691604 1897 heartbeater.cc:503] Master 127.0.113.62:41193 was elected leader, sending a full tablet report...
I20260806 11:41:33.738140 1929 raft_consensus.cc:482] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:33.738372 1929 raft_consensus.cc:504] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } }
I20260806 11:41:33.739511 1929 leader_election.cc:288] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0ccefb31fc4de6a5409090b66adf00 (127.0.113.2:42925), e018388a5f6a4372ab79985aa31207ad (127.0.113.3:33799)
I20260806 11:41:33.744792 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.740409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:53031 (local address 127.0.113.3:33799)
0806 11:41:33.740559 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.740564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.740722 (+ 158us) server_negotiation.cc:407] Connection header received
0806 11:41:33.740836 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.740838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.740875 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.740977 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:33.741242 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.741989 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.743463 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.744086 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.744197 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.744259 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.744588 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.744660 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 11:41:33.744792 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.739885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:56063 (local address 127.0.113.2:42925)
0806 11:41:33.740030 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.740037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.740156 (+ 119us) server_negotiation.cc:407] Connection header received
0806 11:41:33.740541 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.740544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.740590 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.740655 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:33.740972 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.741747 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.743463 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.744086 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.744197 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.744259 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.744588 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.744660 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:41:33.744850 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.740214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:33799 (local address 127.0.113.1:53031)
0806 11:41:33.740598 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:33.740610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:33.740734 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:33.741004 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:33.741006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:33.741022 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:33.741145 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.741149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.742099 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.742102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.743337 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.743343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.744239 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.744243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.744430 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.744482 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 11:41:33.744556 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 11:41:33.744662 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":122,"spinlock_wait_cycles":6272,"thread_start_us":144,"threads_started":1}
I20260806 11:41:33.744853 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.739781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:42925 (local address 127.0.113.1:56063)
0806 11:41:33.740066 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:33.740079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:33.740207 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:33.740694 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:33.740697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:33.740725 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:33.740862 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.740868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.741890 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.741893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.743325 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:33.743337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.744239 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.744243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:33.744434 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.744473 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 11:41:33.744556 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 11:41:33.744662 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":93,"spinlock_wait_cycles":384,"thread_start_us":119,"threads_started":1}
I20260806 11:41:33.745692 1652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c283753528fe4059980f36368ac8fe8b" candidate_uuid: "bbb6ada01dd14acaa3057cc831ae557d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0ccefb31fc4de6a5409090b66adf00" is_pre_election: true
I20260806 11:41:33.746106 1652 raft_consensus.cc:2454] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbb6ada01dd14acaa3057cc831ae557d in term 0.
I20260806 11:41:33.746682 1372 leader_election.cc:302] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0ccefb31fc4de6a5409090b66adf00, bbb6ada01dd14acaa3057cc831ae557d; no voters:
I20260806 11:41:33.746951 1929 raft_consensus.cc:2781] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:33.747051 1929 raft_consensus.cc:482] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:33.747116 1929 raft_consensus.cc:3037] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:33.747478 1829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c283753528fe4059980f36368ac8fe8b" candidate_uuid: "bbb6ada01dd14acaa3057cc831ae557d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e018388a5f6a4372ab79985aa31207ad" is_pre_election: true
I20260806 11:41:33.747766 1829 raft_consensus.cc:2454] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbb6ada01dd14acaa3057cc831ae557d in term 0.
I20260806 11:41:33.748448 1929 raft_consensus.cc:504] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } }
I20260806 11:41:33.748796 1929 leader_election.cc:288] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [CANDIDATE]: Term 1 election: Requested vote from peers 1c0ccefb31fc4de6a5409090b66adf00 (127.0.113.2:42925), e018388a5f6a4372ab79985aa31207ad (127.0.113.3:33799)
I20260806 11:41:33.749208 1652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c283753528fe4059980f36368ac8fe8b" candidate_uuid: "bbb6ada01dd14acaa3057cc831ae557d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0ccefb31fc4de6a5409090b66adf00"
I20260806 11:41:33.749238 1829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c283753528fe4059980f36368ac8fe8b" candidate_uuid: "bbb6ada01dd14acaa3057cc831ae557d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e018388a5f6a4372ab79985aa31207ad"
I20260806 11:41:33.749343 1652 raft_consensus.cc:3037] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:33.749348 1829 raft_consensus.cc:3037] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:33.750377 1652 raft_consensus.cc:2454] T c283753528fe4059980f36368ac8fe8b P 1c0ccefb31fc4de6a5409090b66adf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbb6ada01dd14acaa3057cc831ae557d in term 1.
I20260806 11:41:33.750412 1829 raft_consensus.cc:2454] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbb6ada01dd14acaa3057cc831ae557d in term 1.
I20260806 11:41:33.750670 1372 leader_election.cc:302] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0ccefb31fc4de6a5409090b66adf00, bbb6ada01dd14acaa3057cc831ae557d; no voters:
I20260806 11:41:33.750876 1929 raft_consensus.cc:2781] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:33.751108 1929 raft_consensus.cc:686] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [term 1 LEADER]: Becoming Leader. State: Replica: bbb6ada01dd14acaa3057cc831ae557d, State: Running, Role: LEADER
I20260806 11:41:33.751428 1929 consensus_queue.cc:237] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } }
I20260806 11:41:33.754850 1255 catalog_manager.cc:5495] T c283753528fe4059980f36368ac8fe8b P bbb6ada01dd14acaa3057cc831ae557d reported cstate change: term changed from 0 to 1, leader changed from <none> to bbb6ada01dd14acaa3057cc831ae557d (127.0.113.1). New cstate: current_term: 1 leader_uuid: "bbb6ada01dd14acaa3057cc831ae557d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ccefb31fc4de6a5409090b66adf00" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e018388a5f6a4372ab79985aa31207ad" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbb6ada01dd14acaa3057cc831ae557d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38465 } health_report { overall_health: HEALTHY } } }
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:41:33.787704 1919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.779249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.0.113.1:38465)
0806 11:41:33.779441 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.779444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.779535 (+ 91us) server_negotiation.cc:407] Connection header received
0806 11:41:33.779640 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.779642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.779686 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.779757 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:33.780476 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.780975 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.781769 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.781989 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.782067 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.784553 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:33.784565 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:33.784581 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:33.784605 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:33.786092 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.786501 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.786507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.786508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.786590 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.786827 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.786829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.786830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.787091 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:33.787252 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.787387 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.787484 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:41:33.815977 1919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.808046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.0.113.1:38465)
0806 11:41:33.808178 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.808182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.808382 (+ 200us) server_negotiation.cc:407] Connection header received
0806 11:41:33.808517 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.808520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.808561 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.808648 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:33.809338 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.809857 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.810651 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.810865 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.810937 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.813236 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:33.813239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:33.813241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:33.813265 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:33.814590 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.814973 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.814976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.814977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.815050 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.815273 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.815275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.815276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.815435 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:33.815553 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.815762 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.815855 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:41:33.842777 1919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:33.834750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.0.113.1:38465)
0806 11:41:33.834942 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:33.834946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:33.835056 (+ 110us) server_negotiation.cc:407] Connection header received
0806 11:41:33.835234 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:33.835237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:33.835298 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:33.835387 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:33.836078 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.836650 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.837440 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:33.837664 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:33.837780 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:33.840038 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:33.840041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:33.840043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:33.840064 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:33.841261 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.841652 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.841654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.841656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.841727 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:33.842016 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:33.842019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:33.842020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:33.842174 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:33.842291 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:33.842483 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 11:41:33.842612 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 11:41:33.844491 1407 tablet_service.cc:2908] Rejecting scan request for tablet c283753528fe4059980f36368ac8fe8b: Uninitialized: clean time has not yet been initialized
I20260806 11:41:33.846827 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 1343
W20260806 11:41:33.862561 1898 tablet.cc:2369] T c283753528fe4059980f36368ac8fe8b P e018388a5f6a4372ab79985aa31207ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:41:33.871685 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 1543
I20260806 11:41:33.881901 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 1720
I20260806 11:41:33.899840 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 1213
2026-08-06T11:41:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2563 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1956](info): set up 2 sockets
Aug 06 11:41:33 dist-test-slave-45dx krb5kdc[1956](info): commencing operation
krb5kdc: starting...
W20260806 11:41:35.122444 452 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:35 dist-test-slave-45dx krb5kdc[1956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:35.247555 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33455
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:35.354734 1977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:35.355075 1977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:35.355158 1977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:35.359174 1977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:35.359274 1977 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 11:41:35.359308 1977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:35.359349 1977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:35.359380 1977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:35.359411 1977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:35.364807 1977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:33455
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:35.365927 1977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:35.366974 1977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:35.375656 1982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:35.375597 1984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:35.375674 1986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:35.375901 1977 server_base.cc:1034] running on GCE node
I20260806 11:41:35.376673 1977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:35.377828 1977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:35.379046 1977 hybrid_clock.cc:648] HybridClock initialized: now 1786016495379024 us; error 36 us; skew 500 ppm
Aug 06 11:41:35 dist-test-slave-45dx krb5kdc[1956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:35.381954 1977 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:35.382840 1977 webserver.cc:460] Webserver started at http://127.0.113.62:38413/ using document root <none> and password file <none>
I20260806 11:41:35.383430 1977 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:35.383526 1977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:35.383769 1977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:35.385917 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "a71030befde2405e9722572d1c92b8b4"
format_stamp: "Formatted at 2026-08-06 11:41:35 on dist-test-slave-45dx"
server_key: "72ac7c06849d9af15718f43815e12cb7"
server_key_iv: "f02a9b42c2f0aa280fe972003a1d448b"
server_key_version: "encryptionkey@0"
I20260806 11:41:35.386497 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "a71030befde2405e9722572d1c92b8b4"
format_stamp: "Formatted at 2026-08-06 11:41:35 on dist-test-slave-45dx"
server_key: "72ac7c06849d9af15718f43815e12cb7"
server_key_iv: "f02a9b42c2f0aa280fe972003a1d448b"
server_key_version: "encryptionkey@0"
I20260806 11:41:35.390242 1977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:41:35.392652 1998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:35.393878 1977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:41:35.394037 1977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "a71030befde2405e9722572d1c92b8b4"
format_stamp: "Formatted at 2026-08-06 11:41:35 on dist-test-slave-45dx"
server_key: "72ac7c06849d9af15718f43815e12cb7"
server_key_iv: "f02a9b42c2f0aa280fe972003a1d448b"
server_key_version: "encryptionkey@0"
I20260806 11:41:35.394150 1977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:35.405666 1977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:35.408634 1977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:35.408942 1977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:35.416990 1977 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:33455
I20260806 11:41:35.417065 2087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:33455 every 8 connection(s)
I20260806 11:41:35.417944 1977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:35.420738 2088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:35.423605 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 1977
I20260806 11:41:35.423735 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:35.424021 452 external_mini_cluster.cc:1428] Setting key 5886562caeb7b0db7d32de123fcb069d
I20260806 11:41:35.426697 2088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4: Bootstrap starting.
I20260806 11:41:35.429632 2088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:35.430572 2088 log.cc:824] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:35.432685 2088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4: No bootstrap required, opened a new log
Aug 06 11:41:35 dist-test-slave-45dx krb5kdc[1956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:35.436684 2088 raft_consensus.cc:348] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71030befde2405e9722572d1c92b8b4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33455 } }
I20260806 11:41:35.436967 2088 raft_consensus.cc:374] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:35.437079 2088 raft_consensus.cc:729] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71030befde2405e9722572d1c92b8b4, State: Initialized, Role: FOLLOWER
I20260806 11:41:35.437593 2088 consensus_queue.cc:260] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71030befde2405e9722572d1c92b8b4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33455 } }
I20260806 11:41:35.437788 2088 raft_consensus.cc:388] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:35.437923 2088 raft_consensus.cc:482] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:35.438051 2088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:35.439143 2088 raft_consensus.cc:504] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71030befde2405e9722572d1c92b8b4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33455 } }
I20260806 11:41:35.439391 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:35.425338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.0.113.62:33455)
0806 11:41:35.425764 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:35.425779 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:35.425814 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:41:35.426418 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:35.426433 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:35.426741 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:35.427193 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:35.428107 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.429177 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.430455 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.430975 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.431153 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:35.434294 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:35.434300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:35.434313 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:35.434341 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:35.436195 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.436772 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.436777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.436782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.436917 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.437264 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.437267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.437269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.437709 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:35.437977 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:35.438251 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 11:41:35.438559 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":134,"threads_started":1}
I20260806 11:41:35.439524 2088 leader_election.cc:302] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a71030befde2405e9722572d1c92b8b4; no voters:
I20260806 11:41:35.439965 2088 leader_election.cc:288] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:35.440037 2098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:35.440241 2098 raft_consensus.cc:686] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [term 1 LEADER]: Becoming Leader. State: Replica: a71030befde2405e9722572d1c92b8b4, State: Running, Role: LEADER
I20260806 11:41:35.440634 2098 consensus_queue.cc:237] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71030befde2405e9722572d1c92b8b4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33455 } }
I20260806 11:41:35.441313 2088 sys_catalog.cc:564] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:35.442394 2101 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a71030befde2405e9722572d1c92b8b4. Latest consensus state: current_term: 1 leader_uuid: "a71030befde2405e9722572d1c92b8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71030befde2405e9722572d1c92b8b4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33455 } } }
I20260806 11:41:35.442543 2101 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:35.442948 2109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:35.442937 2100 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a71030befde2405e9722572d1c92b8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71030befde2405e9722572d1c92b8b4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33455 } } }
I20260806 11:41:35.443186 2100 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:35.446053 2109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:35.451282 2109 catalog_manager.cc:1347] Generated new cluster ID: 51242aeffb1f4966ab7d07f5a8f4853b
I20260806 11:41:35.451355 2109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:35.473181 2109 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:35.474138 2109 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:35.484472 2109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4: Generated new TSK 0
I20260806 11:41:35.485308 2109 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:41:35.527782 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33455
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:35.659288 2126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:35.659550 2126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:35.659593 2126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:35.663482 2126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:35.663530 2126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:35.663565 2126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 11:41:35.663595 2126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:35.669126 2126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.113.62:33455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:35.670092 2126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:35.671072 2126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:35.680678 2135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:35.680668 2138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:35.680692 2126 server_base.cc:1034] running on GCE node
W20260806 11:41:35.680668 2133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:35.681424 2126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:35.682013 2126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:35.683178 2126 hybrid_clock.cc:648] HybridClock initialized: now 1786016495683179 us; error 58 us; skew 500 ppm
Aug 06 11:41:35 dist-test-slave-45dx krb5kdc[1956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:35.686048 2126 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:41:35.686955 2126 webserver.cc:460] Webserver started at http://127.0.113.1:43431/ using document root <none> and password file <none>
I20260806 11:41:35.687528 2126 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:35.687593 2126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:35.687839 2126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:35.689965 2126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "df7f2b9ba81a43ea9bb66a1237b7b795"
format_stamp: "Formatted at 2026-08-06 11:41:35 on dist-test-slave-45dx"
server_key: "b6a9a5caeadfc7442e9c84630f3162bd"
server_key_iv: "a145b5ae85332f39fa77ed5efffee399"
server_key_version: "encryptionkey@0"
I20260806 11:41:35.690572 2126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "df7f2b9ba81a43ea9bb66a1237b7b795"
format_stamp: "Formatted at 2026-08-06 11:41:35 on dist-test-slave-45dx"
server_key: "b6a9a5caeadfc7442e9c84630f3162bd"
server_key_iv: "a145b5ae85332f39fa77ed5efffee399"
server_key_version: "encryptionkey@0"
I20260806 11:41:35.694586 2126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 11:41:35.697089 2150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:35.698342 2126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:35.698468 2126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "df7f2b9ba81a43ea9bb66a1237b7b795"
format_stamp: "Formatted at 2026-08-06 11:41:35 on dist-test-slave-45dx"
server_key: "b6a9a5caeadfc7442e9c84630f3162bd"
server_key_iv: "a145b5ae85332f39fa77ed5efffee399"
server_key_version: "encryptionkey@0"
I20260806 11:41:35.698568 2126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:35.734108 2126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:35.737253 2126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:35.737491 2126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:35.738992 2126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:35.739045 2126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:35.739125 2126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:35.739156 2126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 11:41:35 dist-test-slave-45dx krb5kdc[1956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:35.750720 2126 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:45663
I20260806 11:41:35.750818 2316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:45663 every 8 connection(s)
I20260806 11:41:35.751945 2126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:35.753847 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:35.739977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:37181 (local address 127.0.113.62:33455)
0806 11:41:35.740114 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:35.740118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:35.741236 (+ 1118us) server_negotiation.cc:407] Connection header received
0806 11:41:35.742551 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:35.742555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:35.742620 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:35.742745 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:35.744546 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.745169 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.746570 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.746839 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.746956 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:35.749912 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:35.749916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:35.749919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:35.749955 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:35.751698 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.752254 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.752261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.752265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.752413 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.752844 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.752848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.752850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.753043 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:35.753123 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:35.753617 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 11:41:35.753725 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:41:35.754612 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:35.740557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33455 (local address 127.0.113.1:37181)
0806 11:41:35.741070 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:35.741126 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:35.742242 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:35.743003 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:35.743015 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:35.743496 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:35.744372 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:35.744397 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.745324 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.745334 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:35.746430 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:35.746442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.747014 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.747018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:35.747234 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:35.747837 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:35.747854 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:35.749731 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:35.751814 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:35.751819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:35.751829 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:35.752125 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:35.752541 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:35.752544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:35.752546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:35.752732 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:35.753146 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:35.753151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:35.753371 (+ 220us) client_negotiation.cc:769] Sending connection context
0806 11:41:35.753551 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 11:41:35.753793 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":136,"threads_started":1}
I20260806 11:41:35.755717 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 2126
I20260806 11:41:35.755872 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:35.756083 2318 heartbeater.cc:347] Connected to a master server at 127.0.113.62:33455
I20260806 11:41:35.756227 452 external_mini_cluster.cc:1428] Setting key 9c838fe0c0f5ed6e04b6ae49251b4897
I20260806 11:41:35.756249 2318 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:35.756835 2318 heartbeater.cc:511] Master 127.0.113.62:33455 requested a full tablet report, sending...
I20260806 11:41:35.758477 2023 ts_manager.cc:194] Registered new tserver with Master: df7f2b9ba81a43ea9bb66a1237b7b795 (127.0.113.1:45663)
I20260806 11:41:35.759655 2023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:37181
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:41:35.767324 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:35.758611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:44957 (local address 127.0.113.62:33455)
0806 11:41:35.760271 (+ 1660us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:35.760276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:35.760578 (+ 302us) server_negotiation.cc:407] Connection header received
0806 11:41:35.760746 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:35.760749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:35.760789 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:35.760883 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:35.761358 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.761875 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.762690 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.762927 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.763040 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:35.764132 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:35.764136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:35.764139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:35.764162 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:35.765637 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.766093 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.766097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.766099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.766183 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.766468 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.766471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.766473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.766645 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:35.766749 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:35.767065 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 11:41:35.767177 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1558}
I20260806 11:41:35.767340 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:35.758470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33455 (local address 127.0.113.1:44957)
0806 11:41:35.760497 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:35.760513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:35.760647 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:35.760900 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:35.760902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:35.761106 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:35.761232 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:35.761238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.762011 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.762015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:35.762569 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:35.762577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.763046 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.763050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:35.763148 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:35.763676 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:35.763688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:35.763993 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:35.765769 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:35.765773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:35.765775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:35.765983 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:35.766276 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:35.766280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:35.766282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:35.766369 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:35.766753 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:35.766757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:35.766852 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 11:41:35.766989 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 11:41:35.767162 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1914,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:41:35.787350 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33455
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:35.900269 2331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:35.900681 2331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:35.900758 2331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:35.904909 2331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:35.904973 2331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:35.905014 2331 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 11:41:35.905047 2331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:41:35.910277 2331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.113.62:33455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:35.911381 2331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:35.912444 2331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:35.922151 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:35.922147 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:35.922206 2331 server_base.cc:1034] running on GCE node
W20260806 11:41:35.922153 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:35.922757 2331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:35.923450 2331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:35.924633 2331 hybrid_clock.cc:648] HybridClock initialized: now 1786016495924621 us; error 32 us; skew 500 ppm
Aug 06 11:41:35 dist-test-slave-45dx krb5kdc[1956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:35.927366 2331 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:41:35.928218 2331 webserver.cc:460] Webserver started at http://127.0.113.2:46721/ using document root <none> and password file <none>
I20260806 11:41:35.928776 2331 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:35.928870 2331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:35.929105 2331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:35.930998 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "a6214940dc1949fcb60a3a503ff80916"
format_stamp: "Formatted at 2026-08-06 11:41:35 on dist-test-slave-45dx"
server_key: "1fed356bf72631230d58a3d7c44cca06"
server_key_iv: "8d15c0d4081c3006398c53f89e15fcda"
server_key_version: "encryptionkey@0"
I20260806 11:41:35.931623 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "a6214940dc1949fcb60a3a503ff80916"
format_stamp: "Formatted at 2026-08-06 11:41:35 on dist-test-slave-45dx"
server_key: "1fed356bf72631230d58a3d7c44cca06"
server_key_iv: "8d15c0d4081c3006398c53f89e15fcda"
server_key_version: "encryptionkey@0"
I20260806 11:41:35.935179 2331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:41:35.937582 2355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:35.938721 2331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:41:35.938853 2331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "a6214940dc1949fcb60a3a503ff80916"
format_stamp: "Formatted at 2026-08-06 11:41:35 on dist-test-slave-45dx"
server_key: "1fed356bf72631230d58a3d7c44cca06"
server_key_iv: "8d15c0d4081c3006398c53f89e15fcda"
server_key_version: "encryptionkey@0"
I20260806 11:41:35.938963 2331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:35.953043 2331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:35.956219 2331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:35.956460 2331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:35.957939 2331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:35.958012 2331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:35.958081 2331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:35.958137 2331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 11:41:35 dist-test-slave-45dx krb5kdc[1956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:35.969290 2331 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:37195
I20260806 11:41:35.969396 2519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:37195 every 8 connection(s)
I20260806 11:41:35.970304 2331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:41:35.971920 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:35.958625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:56301 (local address 127.0.113.62:33455)
0806 11:41:35.958774 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:35.958779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:35.959646 (+ 867us) server_negotiation.cc:407] Connection header received
0806 11:41:35.960721 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:35.960726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:35.960784 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:35.960846 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:35.962183 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.962742 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.963963 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.964254 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.964418 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:35.968234 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:35.968242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:35.968245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:35.968270 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:35.969920 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.970382 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.970386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.970387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.970479 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.970798 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.970804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.970809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.971037 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:35.971138 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:35.971604 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 11:41:35.971722 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 11:41:35.972654 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:35.959026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33455 (local address 127.0.113.2:56301)
0806 11:41:35.959475 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:35.959522 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:35.960478 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:35.961276 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:35.961281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:35.961574 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:35.962037 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:35.962050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.962901 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.962904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:35.963832 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:35.963842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.964457 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.964461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:35.964705 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:35.965321 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:35.965348 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:35.968064 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:35.970028 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:35.970032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:35.970042 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:35.970275 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:35.970565 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:35.970568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:35.970569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:35.970697 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:35.971143 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:35.971150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:35.971372 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 11:41:35.971545 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 11:41:35.971831 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":131,"threads_started":1}
I20260806 11:41:35.973896 2520 heartbeater.cc:347] Connected to a master server at 127.0.113.62:33455
I20260806 11:41:35.974048 2520 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:35.974524 2520 heartbeater.cc:511] Master 127.0.113.62:33455 requested a full tablet report, sending...
I20260806 11:41:35.974817 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 2331
I20260806 11:41:35.974941 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:41:35.975190 452 external_mini_cluster.cc:1428] Setting key 35c71f41dd0c1b09277289fdee66e02c
I20260806 11:41:35.976567 2024 ts_manager.cc:194] Registered new tserver with Master: a6214940dc1949fcb60a3a503ff80916 (127.0.113.2:37195)
I20260806 11:41:35.977349 2024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:56301
I20260806 11:41:35.983654 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:35.976363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:36667 (local address 127.0.113.62:33455)
0806 11:41:35.976627 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:35.976630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:35.976653 (+ 23us) server_negotiation.cc:407] Connection header received
0806 11:41:35.976679 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:35.976681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:35.976711 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:35.976800 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:35.977203 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.977908 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.978713 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.978951 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.979060 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:35.980416 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:35.980419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:35.980421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:35.980441 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:35.981889 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.982397 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.982399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.982400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.982472 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:35.982776 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:35.982778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:35.982780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:35.982974 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:35.983062 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:35.983396 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 11:41:35.983481 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":201}
I20260806 11:41:35.983777 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:35.975925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33455 (local address 127.0.113.2:36667)
0806 11:41:35.976209 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:35.976225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:35.976350 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:35.976817 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:35.976820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:35.977007 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:35.977094 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:35.977099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.978034 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.978043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:35.978576 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:35.978583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:35.979066 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:35.979070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:35.979176 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:35.979764 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:35.979783 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:35.980237 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:35.982004 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:35.982009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:35.982013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:35.982290 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:35.982570 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:35.982576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:35.982579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:35.982682 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:35.983090 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:35.983096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:35.983214 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 11:41:35.983338 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 11:41:35.983514 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:41:36.004678 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33455
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:36.116554 2533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:36.116834 2533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:36.116940 2533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:36.121330 2533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:36.121438 2533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:36.121516 2533 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 11:41:36.121598 2533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:41:36.127169 2533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.113.62:33455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:36.128232 2533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:36.129313 2533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:36.138973 2545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:36.138978 2539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:36.138995 2540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:36.139326 2533 server_base.cc:1034] running on GCE node
I20260806 11:41:36.139746 2533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:36.140486 2533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:36.141655 2533 hybrid_clock.cc:648] HybridClock initialized: now 1786016496141634 us; error 33 us; skew 500 ppm
Aug 06 11:41:36 dist-test-slave-45dx krb5kdc[1956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:36.144811 2533 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:41:36.145648 2533 webserver.cc:460] Webserver started at http://127.0.113.3:46255/ using document root <none> and password file <none>
I20260806 11:41:36.146204 2533 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:36.146297 2533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:36.146531 2533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:36.148566 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "adb8d6dc6724499b903aa44fcd681a40"
format_stamp: "Formatted at 2026-08-06 11:41:36 on dist-test-slave-45dx"
server_key: "1c35b8fbfe697b02bd1ec5fbbcfaf358"
server_key_iv: "53c7a2314389b04b2379a9a0fb69ccf0"
server_key_version: "encryptionkey@0"
I20260806 11:41:36.149156 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "adb8d6dc6724499b903aa44fcd681a40"
format_stamp: "Formatted at 2026-08-06 11:41:36 on dist-test-slave-45dx"
server_key: "1c35b8fbfe697b02bd1ec5fbbcfaf358"
server_key_iv: "53c7a2314389b04b2379a9a0fb69ccf0"
server_key_version: "encryptionkey@0"
I20260806 11:41:36.152719 2533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:41:36.154971 2558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:36.156140 2533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:36.156275 2533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "adb8d6dc6724499b903aa44fcd681a40"
format_stamp: "Formatted at 2026-08-06 11:41:36 on dist-test-slave-45dx"
server_key: "1c35b8fbfe697b02bd1ec5fbbcfaf358"
server_key_iv: "53c7a2314389b04b2379a9a0fb69ccf0"
server_key_version: "encryptionkey@0"
I20260806 11:41:36.156451 2533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:36.179281 2533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:36.182982 2533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:36.183207 2533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:36.184890 2533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:36.184967 2533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:36.185037 2533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:36.185081 2533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 11:41:36 dist-test-slave-45dx krb5kdc[1956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:36.196602 2533 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:44145
I20260806 11:41:36.196691 2730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:44145 every 8 connection(s)
I20260806 11:41:36.197711 2533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:41:36.201323 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.185807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:45721 (local address 127.0.113.62:33455)
0806 11:41:36.185941 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.185945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.187202 (+ 1257us) server_negotiation.cc:407] Connection header received
0806 11:41:36.188403 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.188410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.188485 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.188567 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:36.190499 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.191404 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.193291 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.193689 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.193824 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.197276 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:36.197280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:36.197282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:36.197321 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:36.199158 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.199684 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.199688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.199690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.199776 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.200145 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.200149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.200151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.200413 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:36.200532 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.201029 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.201150 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:41:36.201965 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 2533
I20260806 11:41:36.202103 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:41:36.202350 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.186393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33455 (local address 127.0.113.3:45721)
0806 11:41:36.186944 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.186988 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.188053 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.189049 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.189061 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.189528 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:36.190314 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.190337 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.191571 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.191577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.193148 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.193159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.193902 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.193906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.194157 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.195071 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:36.195094 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:36.197098 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:36.199281 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:36.199287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:36.199301 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:36.199567 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:36.199874 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:36.199880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:36.199884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:36.200027 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:36.200531 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:36.200538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:36.200793 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.200972 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.201282 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":145,"threads_started":1}
I20260806 11:41:36.202417 452 external_mini_cluster.cc:1428] Setting key 361f92d1d44351289734efd196d0d972
I20260806 11:41:36.203665 2731 heartbeater.cc:347] Connected to a master server at 127.0.113.62:33455
I20260806 11:41:36.203812 2731 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:36.204424 2731 heartbeater.cc:511] Master 127.0.113.62:33455 requested a full tablet report, sending...
I20260806 11:41:36.205559 2021 ts_manager.cc:194] Registered new tserver with Master: adb8d6dc6724499b903aa44fcd681a40 (127.0.113.3:44145)
I20260806 11:41:36.206387 2021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:45721
I20260806 11:41:36.213164 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.206105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33455 (local address 127.0.113.3:44235)
0806 11:41:36.206798 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.206813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.206921 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.207152 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.207154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.207329 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:36.207430 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.207436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.208412 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.208416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.208952 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.208960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.209374 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.209376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.209444 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.209997 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:36.210018 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:36.210341 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:36.211779 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:36.211781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:36.211783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:36.212018 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:36.212260 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:36.212262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:36.212263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:36.212396 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:36.212740 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:36.212742 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:36.212816 (+ 74us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.212913 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.213018 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":97,"threads_started":1}
I20260806 11:41:36.213176 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.206180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:44235 (local address 127.0.113.62:33455)
0806 11:41:36.206303 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.206308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.206908 (+ 600us) server_negotiation.cc:407] Connection header received
0806 11:41:36.207013 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.207016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.207064 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.207145 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:36.207541 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.208251 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.209053 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.209275 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.209353 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.210457 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:36.210460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:36.210462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:36.210479 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:36.211664 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.212104 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.212106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.212107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.212182 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.212477 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.212479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.212480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.212652 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:36.212735 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.212975 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.213067 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 11:41:36.216342 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 11:41:36 dist-test-slave-45dx krb5kdc[1956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 11:41:36 dist-test-slave-45dx krb5kdc[1956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:36.243908 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.235627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.113.62:33455)
0806 11:41:36.235762 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.235766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.235779 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:41:36.235823 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.235825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.235865 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.235939 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:36.236732 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.237238 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.238064 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.238258 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.238353 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.240726 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:36.240729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:36.240731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:36.240756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:36.242253 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.242651 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.242654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.242658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.242787 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.243081 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.243086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.243091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.243321 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:36.243427 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.243669 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.243782 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 11:41:36.246846 2021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:41:36.250054 2021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:36.265254 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.258569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:45663 (local address 127.0.0.1:37044)
0806 11:41:36.259449 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.259462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.259544 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.260435 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.260438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.260458 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:36.260560 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.260566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.262021 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.262025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.263440 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.263450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.264697 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.264700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.264835 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.264874 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.264979 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.265093 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":81,"thread_start_us":103,"threads_started":1}
I20260806 11:41:36.265791 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.258308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:37195 (local address 127.0.0.1:39052)
0806 11:41:36.258981 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.259055 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.259184 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.259822 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.259827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.259849 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:36.260114 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.260122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.261958 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.261963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.263550 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.263565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.265130 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.265136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.265259 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.265289 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.265366 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.265648 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":97,"threads_started":1}
I20260806 11:41:36.266460 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.259046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:44145 (local address 127.0.0.1:55352)
0806 11:41:36.260160 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.260174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.260274 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.260908 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.260911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.260929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:36.261030 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.261036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.263206 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.263209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.264280 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.264332 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.266049 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.266053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.266159 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.266191 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.266265 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.266331 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":111,"threads_started":1}
I20260806 11:41:36.266472 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.258970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.0.113.1:45663)
0806 11:41:36.259255 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.259264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.259619 (+ 355us) server_negotiation.cc:407] Connection header received
0806 11:41:36.259790 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.259795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.260096 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.260255 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:36.260719 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.261876 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.263572 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.264406 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.264497 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.264638 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.265637 (+ 999us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.266340 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":104,"threads_started":1}
I20260806 11:41:36.266480 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.258516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.0.113.2:37195)
0806 11:41:36.258923 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.258937 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.259324 (+ 387us) server_negotiation.cc:407] Connection header received
0806 11:41:36.259428 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.259433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.259635 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.260257 (+ 622us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:36.260426 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.261734 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.263717 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.264716 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.264864 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.265052 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.266215 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.266324 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":107,"threads_started":1}
I20260806 11:41:36.267022 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.259484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.0.113.3:44145)
0806 11:41:36.260014 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.260031 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.260242 (+ 211us) server_negotiation.cc:407] Connection header received
0806 11:41:36.260435 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.260439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.260570 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.260746 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:36.261162 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.263082 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.264464 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.265202 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.265309 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.265983 (+ 674us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.266752 (+ 769us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.266885 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":112,"threads_started":1}
I20260806 11:41:36.269877 2629 tablet_service.cc:1467] Processing CreateTablet for tablet b52cf64929df42499d86b1bb187eea06 (DEFAULT_TABLE table=test-table [id=04428fbd98f64ba087a2c00de707788b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:36.269876 2425 tablet_service.cc:1467] Processing CreateTablet for tablet b52cf64929df42499d86b1bb187eea06 (DEFAULT_TABLE table=test-table [id=04428fbd98f64ba087a2c00de707788b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:36.269910 2224 tablet_service.cc:1467] Processing CreateTablet for tablet b52cf64929df42499d86b1bb187eea06 (DEFAULT_TABLE table=test-table [id=04428fbd98f64ba087a2c00de707788b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:36.270991 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52cf64929df42499d86b1bb187eea06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:36.270996 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52cf64929df42499d86b1bb187eea06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:36.270992 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52cf64929df42499d86b1bb187eea06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:36.277714 2767 tablet_bootstrap.cc:492] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: Bootstrap starting.
I20260806 11:41:36.279230 2769 tablet_bootstrap.cc:492] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: Bootstrap starting.
I20260806 11:41:36.280172 2767 tablet_bootstrap.cc:654] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:36.280373 2771 tablet_bootstrap.cc:492] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: Bootstrap starting.
I20260806 11:41:36.281175 2767 log.cc:824] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:36.281518 2769 tablet_bootstrap.cc:654] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:36.282401 2771 tablet_bootstrap.cc:654] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:36.282923 2769 log.cc:824] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:36.283130 2771 log.cc:824] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:36.283998 2767 tablet_bootstrap.cc:492] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: No bootstrap required, opened a new log
I20260806 11:41:36.284224 2767 ts_tablet_manager.cc:1397] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 11:41:36.284904 2769 tablet_bootstrap.cc:492] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: No bootstrap required, opened a new log
I20260806 11:41:36.285053 2771 tablet_bootstrap.cc:492] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: No bootstrap required, opened a new log
I20260806 11:41:36.285091 2769 ts_tablet_manager.cc:1397] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 11:41:36.285213 2771 ts_tablet_manager.cc:1397] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:41:36.287844 2767 raft_consensus.cc:348] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.288130 2767 raft_consensus.cc:374] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:36.288189 2767 raft_consensus.cc:729] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7f2b9ba81a43ea9bb66a1237b7b795, State: Initialized, Role: FOLLOWER
I20260806 11:41:36.288674 2767 consensus_queue.cc:260] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.288944 2769 raft_consensus.cc:348] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.289184 2769 raft_consensus.cc:374] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:36.289251 2769 raft_consensus.cc:729] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb8d6dc6724499b903aa44fcd681a40, State: Initialized, Role: FOLLOWER
I20260806 11:41:36.289283 2318 heartbeater.cc:503] Master 127.0.113.62:33455 was elected leader, sending a full tablet report...
I20260806 11:41:36.289654 2767 ts_tablet_manager.cc:1428] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:41:36.289644 2769 consensus_queue.cc:260] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.290288 2731 heartbeater.cc:503] Master 127.0.113.62:33455 was elected leader, sending a full tablet report...
I20260806 11:41:36.288942 2771 raft_consensus.cc:348] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.290597 2769 ts_tablet_manager.cc:1428] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 11:41:36.290592 2771 raft_consensus.cc:374] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:36.290649 2771 raft_consensus.cc:729] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6214940dc1949fcb60a3a503ff80916, State: Initialized, Role: FOLLOWER
I20260806 11:41:36.291011 2771 consensus_queue.cc:260] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.291638 2520 heartbeater.cc:503] Master 127.0.113.62:33455 was elected leader, sending a full tablet report...
I20260806 11:41:36.291915 2771 ts_tablet_manager.cc:1428] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 11:41:36.302887 2776 raft_consensus.cc:482] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:36.303073 2776 raft_consensus.cc:504] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.304174 2776 leader_election.cc:288] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6214940dc1949fcb60a3a503ff80916 (127.0.113.2:37195), adb8d6dc6724499b903aa44fcd681a40 (127.0.113.3:44145)
I20260806 11:41:36.308043 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.304556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:55023 (local address 127.0.113.2:37195)
0806 11:41:36.304657 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.304661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.304920 (+ 259us) server_negotiation.cc:407] Connection header received
0806 11:41:36.305059 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.305063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.305117 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.305177 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:36.305530 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.306047 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.307157 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.307507 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.307582 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.307655 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.307851 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.307929 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 11:41:36.308087 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.304493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:33593 (local address 127.0.113.3:44145)
0806 11:41:36.304614 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.304619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.305009 (+ 390us) server_negotiation.cc:407] Connection header received
0806 11:41:36.305112 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.305114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.305151 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.305241 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:36.305593 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.306079 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.307192 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.307565 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.307664 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.307720 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.307915 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.307987 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:41:36.308061 2780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.304408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:37195 (local address 127.0.113.1:55023)
0806 11:41:36.304760 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.304772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.304940 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.305212 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.305216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.305235 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:36.305404 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.305410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.306157 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.306161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.307042 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.307051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.307620 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.307625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.307713 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.307743 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.307812 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.307885 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":73,"thread_start_us":109,"threads_started":1}
I20260806 11:41:36.308090 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.304407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:44145 (local address 127.0.113.1:33593)
0806 11:41:36.304901 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.304918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.305023 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.305266 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.305271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.305287 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:36.305429 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.305433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.306214 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.306216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.307051 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.307059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.307683 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.307686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.307773 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.307788 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.307882 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.307937 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":69,"thread_start_us":72,"threads_started":1}
I20260806 11:41:36.308876 2663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52cf64929df42499d86b1bb187eea06" candidate_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb8d6dc6724499b903aa44fcd681a40" is_pre_election: true
I20260806 11:41:36.308961 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52cf64929df42499d86b1bb187eea06" candidate_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6214940dc1949fcb60a3a503ff80916" is_pre_election: true
I20260806 11:41:36.309162 2663 raft_consensus.cc:2454] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df7f2b9ba81a43ea9bb66a1237b7b795 in term 0.
I20260806 11:41:36.309161 2453 raft_consensus.cc:2454] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df7f2b9ba81a43ea9bb66a1237b7b795 in term 0.
I20260806 11:41:36.309623 2155 leader_election.cc:302] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a6214940dc1949fcb60a3a503ff80916, adb8d6dc6724499b903aa44fcd681a40, df7f2b9ba81a43ea9bb66a1237b7b795; no voters:
I20260806 11:41:36.309886 2776 raft_consensus.cc:2781] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:36.309960 2776 raft_consensus.cc:482] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:36.309989 2776 raft_consensus.cc:3037] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:36.311048 2776 raft_consensus.cc:504] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.311369 2776 leader_election.cc:288] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [CANDIDATE]: Term 1 election: Requested vote from peers a6214940dc1949fcb60a3a503ff80916 (127.0.113.2:37195), adb8d6dc6724499b903aa44fcd681a40 (127.0.113.3:44145)
I20260806 11:41:36.311718 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52cf64929df42499d86b1bb187eea06" candidate_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6214940dc1949fcb60a3a503ff80916"
I20260806 11:41:36.311853 2453 raft_consensus.cc:3037] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:36.312435 2663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52cf64929df42499d86b1bb187eea06" candidate_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb8d6dc6724499b903aa44fcd681a40"
I20260806 11:41:36.312557 2663 raft_consensus.cc:3037] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:36.313133 2453 raft_consensus.cc:2454] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df7f2b9ba81a43ea9bb66a1237b7b795 in term 1.
I20260806 11:41:36.313469 2153 leader_election.cc:302] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6214940dc1949fcb60a3a503ff80916, df7f2b9ba81a43ea9bb66a1237b7b795; no voters:
I20260806 11:41:36.313649 2776 raft_consensus.cc:2781] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:36.313915 2776 raft_consensus.cc:686] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 LEADER]: Becoming Leader. State: Replica: df7f2b9ba81a43ea9bb66a1237b7b795, State: Running, Role: LEADER
I20260806 11:41:36.313951 2663 raft_consensus.cc:2454] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df7f2b9ba81a43ea9bb66a1237b7b795 in term 1.
I20260806 11:41:36.314229 2776 consensus_queue.cc:237] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.317230 2023 catalog_manager.cc:5495] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 reported cstate change: term changed from 0 to 1, leader changed from <none> to df7f2b9ba81a43ea9bb66a1237b7b795 (127.0.113.1). New cstate: current_term: 1 leader_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } health_report { overall_health: UNKNOWN } } }
I20260806 11:41:36.341152 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.337683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.0.113.1:45663)
0806 11:41:36.337824 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.337827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.337841 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:41:36.338031 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.338034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.338091 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.338176 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:36.338421 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.338914 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.339752 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.339960 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.340034 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.340369 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:36.340800 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:36.340890 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.340989 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.341040 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:41:36.348598 2663 raft_consensus.cc:1261] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Refusing update from remote peer df7f2b9ba81a43ea9bb66a1237b7b795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:36.348855 2453 raft_consensus.cc:1261] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Refusing update from remote peer df7f2b9ba81a43ea9bb66a1237b7b795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:36.349432 2776 consensus_queue.cc:1035] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:36.349653 2783 consensus_queue.cc:1035] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:36.371181 2023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:36.371455 2023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:36.374462 2023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4: Sending DeleteTablet for 3 replicas of tablet b52cf64929df42499d86b1bb187eea06
I20260806 11:41:36.375356 2425 tablet_service.cc:1514] Processing DeleteTablet for tablet b52cf64929df42499d86b1bb187eea06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:36 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:39052
I20260806 11:41:36.375371 2224 tablet_service.cc:1514] Processing DeleteTablet for tablet b52cf64929df42499d86b1bb187eea06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:36 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:37044
I20260806 11:41:36.375397 2629 tablet_service.cc:1514] Processing DeleteTablet for tablet b52cf64929df42499d86b1bb187eea06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:36 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:55352
I20260806 11:41:36.375998 2803 tablet_replica.cc:330] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: stopping tablet replica
I20260806 11:41:36.376241 2802 tablet_replica.cc:330] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: stopping tablet replica
I20260806 11:41:36.376365 2803 raft_consensus.cc:2229] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:36.376711 2803 raft_consensus.cc:2258] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:36.376696 2802 raft_consensus.cc:2229] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:36.377100 2802 raft_consensus.cc:2258] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:36.377122 2021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:41:36.377570 2021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:36.377907 2800 tablet_replica.cc:330] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: stopping tablet replica
I20260806 11:41:36.378778 2803 ts_tablet_manager.cc:1905] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:36.381263 2802 ts_tablet_manager.cc:1905] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:36.381882 2803 ts_tablet_manager.cc:1918] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:41:36.381997 2803 log.cc:1196] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/b52cf64929df42499d86b1bb187eea06
I20260806 11:41:36.382342 2803 ts_tablet_manager.cc:1939] T b52cf64929df42499d86b1bb187eea06 P a6214940dc1949fcb60a3a503ff80916: Deleting consensus metadata
I20260806 11:41:36.382684 2425 tablet_service.cc:1467] Processing CreateTablet for tablet 751ef073ddeb46f6bf1510c25932c330 (DEFAULT_TABLE table=test-table [id=a921180303654c849768aef4601acfec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:36.382972 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751ef073ddeb46f6bf1510c25932c330. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:36.383586 1999 catalog_manager.cc:4841] TS a6214940dc1949fcb60a3a503ff80916 (127.0.113.2:37195): tablet b52cf64929df42499d86b1bb187eea06 (table test-table [id=04428fbd98f64ba087a2c00de707788b]) successfully deleted
I20260806 11:41:36.383863 2800 raft_consensus.cc:2229] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:41:36.384264 2800 raft_consensus.cc:2258] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:36.385425 2771 tablet_bootstrap.cc:492] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: Bootstrap starting.
I20260806 11:41:36.385490 2224 tablet_service.cc:1467] Processing CreateTablet for tablet 751ef073ddeb46f6bf1510c25932c330 (DEFAULT_TABLE table=test-table [id=a921180303654c849768aef4601acfec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:36.385732 2802 ts_tablet_manager.cc:1918] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:41:36.385753 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751ef073ddeb46f6bf1510c25932c330. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:36.385905 2802 log.cc:1196] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/b52cf64929df42499d86b1bb187eea06
I20260806 11:41:36.386279 2802 ts_tablet_manager.cc:1939] T b52cf64929df42499d86b1bb187eea06 P adb8d6dc6724499b903aa44fcd681a40: Deleting consensus metadata
I20260806 11:41:36.386430 2771 tablet_bootstrap.cc:654] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:36.387539 2001 catalog_manager.cc:4841] TS adb8d6dc6724499b903aa44fcd681a40 (127.0.113.3:44145): tablet b52cf64929df42499d86b1bb187eea06 (table test-table [id=04428fbd98f64ba087a2c00de707788b]) successfully deleted
I20260806 11:41:36.388068 2771 tablet_bootstrap.cc:492] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: No bootstrap required, opened a new log
I20260806 11:41:36.388170 2771 ts_tablet_manager.cc:1397] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 11:41:36.388545 2767 tablet_bootstrap.cc:492] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: Bootstrap starting.
I20260806 11:41:36.388700 2771 raft_consensus.cc:348] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.388867 2771 raft_consensus.cc:374] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:36.388827 2629 tablet_service.cc:1467] Processing CreateTablet for tablet 751ef073ddeb46f6bf1510c25932c330 (DEFAULT_TABLE table=test-table [id=a921180303654c849768aef4601acfec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:36.388902 2771 raft_consensus.cc:729] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6214940dc1949fcb60a3a503ff80916, State: Initialized, Role: FOLLOWER
I20260806 11:41:36.389096 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751ef073ddeb46f6bf1510c25932c330. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:36.389035 2771 consensus_queue.cc:260] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.389302 2771 ts_tablet_manager.cc:1428] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 11:41:36.389528 2767 tablet_bootstrap.cc:654] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:36.390898 2767 tablet_bootstrap.cc:492] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: No bootstrap required, opened a new log
I20260806 11:41:36.390980 2767 ts_tablet_manager.cc:1397] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:36.391467 2769 tablet_bootstrap.cc:492] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: Bootstrap starting.
I20260806 11:41:36.391428 2767 raft_consensus.cc:348] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.391530 2767 raft_consensus.cc:374] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:36.391559 2767 raft_consensus.cc:729] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7f2b9ba81a43ea9bb66a1237b7b795, State: Initialized, Role: FOLLOWER
I20260806 11:41:36.391672 2767 consensus_queue.cc:260] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.391930 2767 ts_tablet_manager.cc:1428] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:36.392515 2769 tablet_bootstrap.cc:654] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:36.393792 2769 tablet_bootstrap.cc:492] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: No bootstrap required, opened a new log
I20260806 11:41:36.393888 2769 ts_tablet_manager.cc:1397] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:36.394301 2769 raft_consensus.cc:348] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.394399 2769 raft_consensus.cc:374] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:36.394431 2769 raft_consensus.cc:729] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb8d6dc6724499b903aa44fcd681a40, State: Initialized, Role: FOLLOWER
I20260806 11:41:36.394546 2769 consensus_queue.cc:260] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.394768 2769 ts_tablet_manager.cc:1428] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:36.395915 2800 ts_tablet_manager.cc:1905] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:36.398474 2800 ts_tablet_manager.cc:1918] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:41:36.398640 2800 log.cc:1196] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/b52cf64929df42499d86b1bb187eea06
I20260806 11:41:36.398963 2800 ts_tablet_manager.cc:1939] T b52cf64929df42499d86b1bb187eea06 P df7f2b9ba81a43ea9bb66a1237b7b795: Deleting consensus metadata
I20260806 11:41:36.400194 2000 catalog_manager.cc:4841] TS df7f2b9ba81a43ea9bb66a1237b7b795 (127.0.113.1:45663): tablet b52cf64929df42499d86b1bb187eea06 (table test-table [id=04428fbd98f64ba087a2c00de707788b]) successfully deleted
W20260806 11:41:36.449180 2732 tablet.cc:2369] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:41:36.471517 2523 tablet.cc:2369] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:41:36.503340 2319 tablet.cc:2369] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:41:36.624116 2794 raft_consensus.cc:482] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:36.624343 2794 raft_consensus.cc:504] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.624918 2794 leader_election.cc:288] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6214940dc1949fcb60a3a503ff80916 (127.0.113.2:37195), adb8d6dc6724499b903aa44fcd681a40 (127.0.113.3:44145)
I20260806 11:41:36.625437 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751ef073ddeb46f6bf1510c25932c330" candidate_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6214940dc1949fcb60a3a503ff80916" is_pre_election: true
I20260806 11:41:36.625448 2663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751ef073ddeb46f6bf1510c25932c330" candidate_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb8d6dc6724499b903aa44fcd681a40" is_pre_election: true
I20260806 11:41:36.625623 2663 raft_consensus.cc:2454] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df7f2b9ba81a43ea9bb66a1237b7b795 in term 0.
I20260806 11:41:36.625624 2453 raft_consensus.cc:2454] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df7f2b9ba81a43ea9bb66a1237b7b795 in term 0.
I20260806 11:41:36.625993 2155 leader_election.cc:302] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adb8d6dc6724499b903aa44fcd681a40, df7f2b9ba81a43ea9bb66a1237b7b795; no voters:
I20260806 11:41:36.626173 2794 raft_consensus.cc:2781] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:36.626240 2794 raft_consensus.cc:482] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:36.626327 2794 raft_consensus.cc:3037] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:36.627418 2794 raft_consensus.cc:504] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.627784 2794 leader_election.cc:288] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [CANDIDATE]: Term 1 election: Requested vote from peers a6214940dc1949fcb60a3a503ff80916 (127.0.113.2:37195), adb8d6dc6724499b903aa44fcd681a40 (127.0.113.3:44145)
I20260806 11:41:36.628152 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751ef073ddeb46f6bf1510c25932c330" candidate_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6214940dc1949fcb60a3a503ff80916"
I20260806 11:41:36.628166 2663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751ef073ddeb46f6bf1510c25932c330" candidate_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb8d6dc6724499b903aa44fcd681a40"
I20260806 11:41:36.628284 2453 raft_consensus.cc:3037] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:36.628284 2663 raft_consensus.cc:3037] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:36.629487 2663 raft_consensus.cc:2454] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df7f2b9ba81a43ea9bb66a1237b7b795 in term 1.
I20260806 11:41:36.629479 2453 raft_consensus.cc:2454] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df7f2b9ba81a43ea9bb66a1237b7b795 in term 1.
I20260806 11:41:36.629784 2153 leader_election.cc:302] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6214940dc1949fcb60a3a503ff80916, df7f2b9ba81a43ea9bb66a1237b7b795; no voters:
I20260806 11:41:36.629988 2794 raft_consensus.cc:2781] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:36.630074 2794 raft_consensus.cc:686] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 LEADER]: Becoming Leader. State: Replica: df7f2b9ba81a43ea9bb66a1237b7b795, State: Running, Role: LEADER
I20260806 11:41:36.630223 2794 consensus_queue.cc:237] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.631836 2021 catalog_manager.cc:5495] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 reported cstate change: term changed from 0 to 1, leader changed from <none> to df7f2b9ba81a43ea9bb66a1237b7b795 (127.0.113.1). New cstate: current_term: 1 leader_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } health_report { overall_health: UNKNOWN } } }
Aug 06 11:41:36 dist-test-slave-45dx krb5kdc[1956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:36.710067 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.702037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.0.113.62:33455)
0806 11:41:36.702147 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.702151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.702412 (+ 261us) server_negotiation.cc:407] Connection header received
0806 11:41:36.702565 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.702570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.702615 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.702701 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:36.703421 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.704033 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.704863 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.705075 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.705185 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.707364 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:36.707367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:36.707369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:36.707391 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:36.708626 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.709045 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.709048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.709049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.709122 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.709355 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.709357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.709358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.709524 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:36.709595 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.709854 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.709944 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 11:41:36.710072 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.701991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33455 (local address 127.0.0.1:60690)
0806 11:41:36.702323 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.702335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.702468 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.702718 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.702721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.703212 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:36.703301 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.703307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.704153 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.704156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.704717 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.704725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.705185 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.705188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.705264 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.705793 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:36.705805 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:36.707236 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:36.708720 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:36.708728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:36.708733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:36.708961 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:36.709199 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:36.709201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:36.709203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:36.709280 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:36.709609 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:36.709614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:36.709710 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.709796 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.709897 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":157,"threads_started":1}
I20260806 11:41:36.714188 2021 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:60690:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 11:41:36.714975 2021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:36.719835 2224 tablet_service.cc:1467] Processing CreateTablet for tablet e7c11f8206f5432cb62e102d812e3e85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d622afa13d3f46378d5273bbc16a8aae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 11:41:36.719985 2629 tablet_service.cc:1467] Processing CreateTablet for tablet e7c11f8206f5432cb62e102d812e3e85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d622afa13d3f46378d5273bbc16a8aae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 11:41:36.720299 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7c11f8206f5432cb62e102d812e3e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:36.720434 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7c11f8206f5432cb62e102d812e3e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:36.719861 2425 tablet_service.cc:1467] Processing CreateTablet for tablet e7c11f8206f5432cb62e102d812e3e85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d622afa13d3f46378d5273bbc16a8aae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 11:41:36.722340 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7c11f8206f5432cb62e102d812e3e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:36.722878 2767 tablet_bootstrap.cc:492] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: Bootstrap starting.
I20260806 11:41:36.722978 2769 tablet_bootstrap.cc:492] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: Bootstrap starting.
I20260806 11:41:36.723801 2769 tablet_bootstrap.cc:654] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:36.723799 2767 tablet_bootstrap.cc:654] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:36.725056 2771 tablet_bootstrap.cc:492] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: Bootstrap starting.
I20260806 11:41:36.725140 2767 tablet_bootstrap.cc:492] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: No bootstrap required, opened a new log
I20260806 11:41:36.725207 2767 ts_tablet_manager.cc:1397] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:36.725294 2769 tablet_bootstrap.cc:492] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: No bootstrap required, opened a new log
I20260806 11:41:36.725375 2769 ts_tablet_manager.cc:1397] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:36.725649 2767 raft_consensus.cc:348] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.725759 2767 raft_consensus.cc:374] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:36.725791 2767 raft_consensus.cc:729] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7f2b9ba81a43ea9bb66a1237b7b795, State: Initialized, Role: FOLLOWER
I20260806 11:41:36.725835 2769 raft_consensus.cc:348] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.725970 2769 raft_consensus.cc:374] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:36.725917 2767 consensus_queue.cc:260] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.726007 2769 raft_consensus.cc:729] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb8d6dc6724499b903aa44fcd681a40, State: Initialized, Role: FOLLOWER
I20260806 11:41:36.726046 2771 tablet_bootstrap.cc:654] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:36.726132 2769 consensus_queue.cc:260] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.726269 2767 ts_tablet_manager.cc:1428] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 11:41:36.726243 2794 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.726372 2794 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:36.726418 2769 ts_tablet_manager.cc:1428] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:36.726619 2778 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.726763 2778 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:36.727494 2771 tablet_bootstrap.cc:492] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: No bootstrap required, opened a new log
I20260806 11:41:36.727576 2771 ts_tablet_manager.cc:1397] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 11:41:36.727988 2771 raft_consensus.cc:348] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.728098 2771 raft_consensus.cc:374] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:36.728140 2771 raft_consensus.cc:729] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6214940dc1949fcb60a3a503ff80916, State: Initialized, Role: FOLLOWER
I20260806 11:41:36.728269 2771 consensus_queue.cc:260] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.728617 2779 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.728765 2779 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:36.728603 2771 ts_tablet_manager.cc:1428] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:36.808082 2779 raft_consensus.cc:482] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:36.808279 2779 raft_consensus.cc:504] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.809536 2779 leader_election.cc:288] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df7f2b9ba81a43ea9bb66a1237b7b795 (127.0.113.1:45663), adb8d6dc6724499b903aa44fcd681a40 (127.0.113.3:44145)
I20260806 11:41:36.813539 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.809917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:40035 (local address 127.0.113.1:45663)
0806 11:41:36.810053 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.810058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.810353 (+ 295us) server_negotiation.cc:407] Connection header received
0806 11:41:36.810508 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.810511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.810558 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.810634 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:36.810925 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.811398 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.812487 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.812905 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.812989 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.813070 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.813298 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.813399 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:41:36.813565 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.810006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:51953 (local address 127.0.113.3:44145)
0806 11:41:36.810229 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.810253 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.810272 (+ 19us) server_negotiation.cc:407] Connection header received
0806 11:41:36.810338 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.810341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.810389 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.810480 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:36.810803 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.811382 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.812555 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.812991 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.813112 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.813175 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.813379 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.813450 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":62,"threads_started":1}
I20260806 11:41:36.813593 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.809814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:44145 (local address 127.0.113.2:51953)
0806 11:41:36.810136 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.810158 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.810262 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.810503 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.810506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.810524 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:36.810672 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.810678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.811498 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.811501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.812412 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.812439 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.813127 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.813130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.813212 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.813228 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.813340 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.813413 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":76,"thread_start_us":71,"threads_started":1}
I20260806 11:41:36.813565 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.809814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:45663 (local address 127.0.113.2:40035)
0806 11:41:36.810217 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.810250 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.810364 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.810654 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.810657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.810671 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:36.810806 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.810813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.811552 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.811554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.812356 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.812365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.813031 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.813036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.813158 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.813184 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.813260 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.813388 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":47,"thread_start_us":87,"threads_started":1}
I20260806 11:41:36.814091 2663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7c11f8206f5432cb62e102d812e3e85" candidate_uuid: "a6214940dc1949fcb60a3a503ff80916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb8d6dc6724499b903aa44fcd681a40" is_pre_election: true
I20260806 11:41:36.814260 2663 raft_consensus.cc:2454] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6214940dc1949fcb60a3a503ff80916 in term 0.
I20260806 11:41:36.814320 2251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7c11f8206f5432cb62e102d812e3e85" candidate_uuid: "a6214940dc1949fcb60a3a503ff80916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" is_pre_election: true
I20260806 11:41:36.814586 2251 raft_consensus.cc:2454] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6214940dc1949fcb60a3a503ff80916 in term 0.
I20260806 11:41:36.814666 2360 leader_election.cc:302] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6214940dc1949fcb60a3a503ff80916, adb8d6dc6724499b903aa44fcd681a40; no voters:
I20260806 11:41:36.814980 2779 raft_consensus.cc:2781] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:36.815047 2779 raft_consensus.cc:482] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:36.815086 2779 raft_consensus.cc:3037] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:36.816115 2779 raft_consensus.cc:504] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.816519 2779 leader_election.cc:288] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [CANDIDATE]: Term 1 election: Requested vote from peers df7f2b9ba81a43ea9bb66a1237b7b795 (127.0.113.1:45663), adb8d6dc6724499b903aa44fcd681a40 (127.0.113.3:44145)
I20260806 11:41:36.816840 2251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7c11f8206f5432cb62e102d812e3e85" candidate_uuid: "a6214940dc1949fcb60a3a503ff80916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795"
I20260806 11:41:36.816906 2663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7c11f8206f5432cb62e102d812e3e85" candidate_uuid: "a6214940dc1949fcb60a3a503ff80916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb8d6dc6724499b903aa44fcd681a40"
I20260806 11:41:36.816977 2251 raft_consensus.cc:3037] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:36.817039 2663 raft_consensus.cc:3037] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:36.817912 2251 raft_consensus.cc:2454] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6214940dc1949fcb60a3a503ff80916 in term 1.
I20260806 11:41:36.817945 2663 raft_consensus.cc:2454] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6214940dc1949fcb60a3a503ff80916 in term 1.
I20260806 11:41:36.818202 2357 leader_election.cc:302] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6214940dc1949fcb60a3a503ff80916, df7f2b9ba81a43ea9bb66a1237b7b795; no voters:
I20260806 11:41:36.818390 2779 raft_consensus.cc:2781] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:36.818572 2779 raft_consensus.cc:686] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [term 1 LEADER]: Becoming Leader. State: Replica: a6214940dc1949fcb60a3a503ff80916, State: Running, Role: LEADER
I20260806 11:41:36.818812 2779 consensus_queue.cc:237] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } }
I20260806 11:41:36.820225 2829 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: TxnStatusTablet state changed. Reason: New leader a6214940dc1949fcb60a3a503ff80916. Latest consensus state: current_term: 1 leader_uuid: "a6214940dc1949fcb60a3a503ff80916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.820366 2829 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: This TxnStatusTablet replica's current role is: LEADER
I20260806 11:41:36.820713 2834 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 11:41:36.820933 2834 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 11:41:36.821717 2021 catalog_manager.cc:5495] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6214940dc1949fcb60a3a503ff80916 (127.0.113.2). New cstate: current_term: 1 leader_uuid: "a6214940dc1949fcb60a3a503ff80916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } health_report { overall_health: UNKNOWN } } }
I20260806 11:41:36.855773 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.851990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.0.113.2:37195)
0806 11:41:36.852396 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.852404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.852415 (+ 11us) server_negotiation.cc:407] Connection header received
0806 11:41:36.852473 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.852475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.852523 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.852642 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:36.852912 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.853420 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.854237 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.854477 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.854584 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.854887 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:36.855285 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:36.855415 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.855495 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.855613 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":136,"threads_started":1}
I20260806 11:41:36.855759 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.851881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:37195 (local address 127.0.0.1:39068)
0806 11:41:36.852016 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.852033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.852141 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.852655 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.852658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.852668 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:41:36.852778 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.852783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.853530 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.853535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.854119 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.854126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.854596 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.854600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.854682 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.854724 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:41:36.855389 (+ 665us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:41:36.855405 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.855500 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.855612 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 11:41:36.858901 2663 raft_consensus.cc:1261] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Refusing update from remote peer a6214940dc1949fcb60a3a503ff80916: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:36.859022 2251 raft_consensus.cc:1261] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 FOLLOWER]: Refusing update from remote peer a6214940dc1949fcb60a3a503ff80916: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:36.859392 2779 consensus_queue.cc:1035] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:36.859767 2779 consensus_queue.cc:1035] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:36.861066 2778 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: TxnStatusTablet state changed. Reason: New leader a6214940dc1949fcb60a3a503ff80916. Latest consensus state: current_term: 1 leader_uuid: "a6214940dc1949fcb60a3a503ff80916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.861191 2778 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:36.861580 2794 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: TxnStatusTablet state changed. Reason: New leader a6214940dc1949fcb60a3a503ff80916. Latest consensus state: current_term: 1 leader_uuid: "a6214940dc1949fcb60a3a503ff80916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.861690 2794 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:36.864267 2829 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6214940dc1949fcb60a3a503ff80916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.864295 2794 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6214940dc1949fcb60a3a503ff80916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.864393 2829 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: This TxnStatusTablet replica's current role is: LEADER
I20260806 11:41:36.864423 2794 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P df7f2b9ba81a43ea9bb66a1237b7b795: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:36.864703 2829 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6214940dc1949fcb60a3a503ff80916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.864792 2829 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P a6214940dc1949fcb60a3a503ff80916: This TxnStatusTablet replica's current role is: LEADER
I20260806 11:41:36.864936 2778 tablet_replica.cc:439] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6214940dc1949fcb60a3a503ff80916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7f2b9ba81a43ea9bb66a1237b7b795" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45663 } } peers { permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 } } peers { permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 } } }
I20260806 11:41:36.865052 2778 tablet_replica.cc:442] T e7c11f8206f5432cb62e102d812e3e85 P adb8d6dc6724499b903aa44fcd681a40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:36.864951 2840 mvcc.cc:205] Tried to move back new op lower bound from 7315523571129061376 to 7315523570972233728. Current Snapshot: MvccSnapshot[applied={T|T < 7315523571129061376}]
I20260806 11:41:36.879802 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.876347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:50243 (local address 127.0.113.2:37195)
0806 11:41:36.876501 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.876504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.876617 (+ 113us) server_negotiation.cc:407] Connection header received
0806 11:41:36.876743 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.876745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.876783 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.876844 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:36.877100 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.877579 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.878818 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.879025 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.879104 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.879433 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:36.879494 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:36.879542 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.879662 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.879703 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 11:41:36.879884 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.876195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:37195 (local address 127.0.113.1:50243)
0806 11:41:36.876528 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.876544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.876655 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.876883 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.876886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.876894 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:41:36.877002 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.877007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.877679 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.877682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.878707 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.878719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.879133 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.879137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.879234 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.879283 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:41:36.879570 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:41:36.879577 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.879645 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.879728 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":123,"threads_started":1}
I20260806 11:41:36.886349 2453 raft_consensus.cc:1261] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Refusing update from remote peer df7f2b9ba81a43ea9bb66a1237b7b795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:36.886366 2663 raft_consensus.cc:1261] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Refusing update from remote peer df7f2b9ba81a43ea9bb66a1237b7b795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:36.886791 2794 consensus_queue.cc:1035] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6214940dc1949fcb60a3a503ff80916" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:36.886989 2783 consensus_queue.cc:1035] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adb8d6dc6724499b903aa44fcd681a40" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:36.890450 2842 mvcc.cc:205] Tried to move back new op lower bound from 7315523571243065344 to 7315523570198659072. Current Snapshot: MvccSnapshot[applied={T|T < 7315523571243065344}]
I20260806 11:41:36.900573 2790 mvcc.cc:205] Tried to move back new op lower bound from 7315523571243065344 to 7315523570198659072. Current Snapshot: MvccSnapshot[applied={T|T < 7315523571243065344}]
I20260806 11:41:36.909519 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.905936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:46121 (local address 127.0.113.1:45663)
0806 11:41:36.906062 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.906065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.906074 (+ 9us) server_negotiation.cc:407] Connection header received
0806 11:41:36.906107 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.906109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.906144 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.906227 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:36.906489 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.907200 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.908467 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.908670 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.908748 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.909075 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:36.909135 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:36.909205 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.909329 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.909382 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 11:41:36.909536 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.905240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:45663 (local address 127.0.113.2:46121)
0806 11:41:36.905791 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:36.905808 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:36.905908 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:36.906236 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:36.906239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:36.906247 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:41:36.906388 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.906398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.907328 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.907331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.908348 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:36.908371 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.908777 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.908780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:36.908901 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.908954 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:41:36.909232 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:41:36.909238 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 11:41:36.909303 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 11:41:36.909379 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
I20260806 11:41:36.932025 2024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:36.932204 2024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:36.934350 2024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a71030befde2405e9722572d1c92b8b4: Sending DeleteTablet for 3 replicas of tablet 751ef073ddeb46f6bf1510c25932c330
I20260806 11:41:36.934862 2224 tablet_service.cc:1514] Processing DeleteTablet for tablet 751ef073ddeb46f6bf1510c25932c330 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:36 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:37044
I20260806 11:41:36.934963 2425 tablet_service.cc:1514] Processing DeleteTablet for tablet 751ef073ddeb46f6bf1510c25932c330 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:36 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:39052
I20260806 11:41:36.935132 2629 tablet_service.cc:1514] Processing DeleteTablet for tablet 751ef073ddeb46f6bf1510c25932c330 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:36 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:55352
I20260806 11:41:36.935179 2866 tablet_replica.cc:330] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: stopping tablet replica
I20260806 11:41:36.935297 2866 raft_consensus.cc:2229] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:41:36.935472 2869 tablet_replica.cc:330] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: stopping tablet replica
I20260806 11:41:36.935492 2866 raft_consensus.cc:2258] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:36.935593 2869 raft_consensus.cc:2229] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:36.935693 2869 raft_consensus.cc:2258] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:36.936187 2866 ts_tablet_manager.cc:1905] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:36.936518 2869 ts_tablet_manager.cc:1905] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:36.937532 2867 tablet_replica.cc:330] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: stopping tablet replica
I20260806 11:41:36.937723 2867 raft_consensus.cc:2229] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:36.937887 2867 raft_consensus.cc:2258] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:36.938545 2867 ts_tablet_manager.cc:1905] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:36.939600 2866 ts_tablet_manager.cc:1918] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:41:36.939692 2866 log.cc:1196] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/751ef073ddeb46f6bf1510c25932c330
I20260806 11:41:36.939738 2869 ts_tablet_manager.cc:1918] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:41:36.939826 2869 log.cc:1196] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/751ef073ddeb46f6bf1510c25932c330
I20260806 11:41:36.940008 2866 ts_tablet_manager.cc:1939] T 751ef073ddeb46f6bf1510c25932c330 P df7f2b9ba81a43ea9bb66a1237b7b795: Deleting consensus metadata
I20260806 11:41:36.940138 2869 ts_tablet_manager.cc:1939] T 751ef073ddeb46f6bf1510c25932c330 P adb8d6dc6724499b903aa44fcd681a40: Deleting consensus metadata
I20260806 11:41:36.940763 2000 catalog_manager.cc:4841] TS df7f2b9ba81a43ea9bb66a1237b7b795 (127.0.113.1:45663): tablet 751ef073ddeb46f6bf1510c25932c330 (table test-table [id=a921180303654c849768aef4601acfec]) successfully deleted
I20260806 11:41:36.940832 2001 catalog_manager.cc:4841] TS adb8d6dc6724499b903aa44fcd681a40 (127.0.113.3:44145): tablet 751ef073ddeb46f6bf1510c25932c330 (table test-table [id=a921180303654c849768aef4601acfec]) successfully deleted
I20260806 11:41:36.942456 2867 ts_tablet_manager.cc:1918] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:41:36.942543 2867 log.cc:1196] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/751ef073ddeb46f6bf1510c25932c330
I20260806 11:41:36.942830 2867 ts_tablet_manager.cc:1939] T 751ef073ddeb46f6bf1510c25932c330 P a6214940dc1949fcb60a3a503ff80916: Deleting consensus metadata
I20260806 11:41:36.943452 1999 catalog_manager.cc:4841] TS a6214940dc1949fcb60a3a503ff80916 (127.0.113.2:37195): tablet 751ef073ddeb46f6bf1510c25932c330 (table test-table [id=a921180303654c849768aef4601acfec]) successfully deleted
Aug 06 11:41:36 dist-test-slave-45dx krb5kdc[1956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:41:36.950644 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.937267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.0.113.1:45663)
0806 11:41:36.937383 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.937388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.940863 (+ 3475us) server_negotiation.cc:407] Connection header received
0806 11:41:36.941084 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.941087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.941138 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.941190 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:36.942342 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.943157 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.944080 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.944366 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.944483 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.947044 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:36.947055 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:36.947071 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:36.947104 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:36.948768 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.949209 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.949215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.949217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.949314 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.949601 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.949604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.949606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.949857 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:36.950074 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.950308 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.950428 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 11:41:36.951150 2307 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37054
I20260806 11:41:36.961092 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:36.952986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.0.113.62:33455)
0806 11:41:36.953115 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:36.953120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:36.953201 (+ 81us) server_negotiation.cc:407] Connection header received
0806 11:41:36.953274 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:36.953277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:36.953323 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:36.953410 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:36.953948 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.954674 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.955466 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:36.955808 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:36.955938 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:36.957424 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:36.957428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:36.957431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:36.957457 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:36.959085 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.959638 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.959641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.959643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.959734 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:36.960124 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:36.960127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:36.960128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:36.960425 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:36.960527 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:36.960800 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 11:41:36.960977 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 11:41:36.961515 2024 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60696
I20260806 11:41:36.962674 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 2126
I20260806 11:41:36.989136 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 2331
I20260806 11:41:36.999979 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 2533
I20260806 11:41:37.010563 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 1977
2026-08-06T11:41:37Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3108 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:37 dist-test-slave-45dx krb5kdc[2879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:37 dist-test-slave-45dx krb5kdc[2879](info): set up 2 sockets
Aug 06 11:41:37 dist-test-slave-45dx krb5kdc[2879](info): commencing operation
krb5kdc: starting...
W20260806 11:41:38.228492 452 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:38 dist-test-slave-45dx krb5kdc[2879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:38.351094 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:37385
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:37385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:38.458171 2900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:38.458429 2900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:38.458489 2900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:38.462522 2900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:38.462580 2900 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 11:41:38.462597 2900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:38.462615 2900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:38.462630 2900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:38.462646 2900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:38.467811 2900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:37385
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:37385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:38.468801 2900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:38.469762 2900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:38.478209 2910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:38.478205 2905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:38.478201 2907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:38.478459 2900 server_base.cc:1034] running on GCE node
I20260806 11:41:38.479133 2900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:38.480214 2900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:38.481448 2900 hybrid_clock.cc:648] HybridClock initialized: now 1786016498481425 us; error 38 us; skew 500 ppm
Aug 06 11:41:38 dist-test-slave-45dx krb5kdc[2879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:38.490871 2900 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:38.491730 2900 webserver.cc:460] Webserver started at http://127.0.113.62:43847/ using document root <none> and password file <none>
I20260806 11:41:38.492281 2900 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:38.492435 2900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:38.492717 2900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:38.494716 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "75743732347a4b06bdd4c017310bc74b"
format_stamp: "Formatted at 2026-08-06 11:41:38 on dist-test-slave-45dx"
server_key: "161bde9678bc4b822b09ff3e46078ba1"
server_key_iv: "b388b871a20f8278f3eb1682fc6784c3"
server_key_version: "encryptionkey@0"
I20260806 11:41:38.495268 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "75743732347a4b06bdd4c017310bc74b"
format_stamp: "Formatted at 2026-08-06 11:41:38 on dist-test-slave-45dx"
server_key: "161bde9678bc4b822b09ff3e46078ba1"
server_key_iv: "b388b871a20f8278f3eb1682fc6784c3"
server_key_version: "encryptionkey@0"
I20260806 11:41:38.499171 2900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 11:41:38.501566 2920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:38.502763 2900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:41:38.502909 2900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "75743732347a4b06bdd4c017310bc74b"
format_stamp: "Formatted at 2026-08-06 11:41:38 on dist-test-slave-45dx"
server_key: "161bde9678bc4b822b09ff3e46078ba1"
server_key_iv: "b388b871a20f8278f3eb1682fc6784c3"
server_key_version: "encryptionkey@0"
I20260806 11:41:38.503036 2900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:38.537058 2900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:38.540428 2900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:38.540627 2900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:38.548671 2900 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:37385
I20260806 11:41:38.548691 3004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:37385 every 8 connection(s)
I20260806 11:41:38.549577 2900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:38.552229 3005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:38.557215 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 2900
I20260806 11:41:38.557308 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:38.557574 452 external_mini_cluster.cc:1428] Setting key 3c31f4bc529661a80123d5146c2da18b
I20260806 11:41:38.557585 3005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: Bootstrap starting.
I20260806 11:41:38.560118 3005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:38.561017 3005 log.cc:824] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:38.563223 3005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: No bootstrap required, opened a new log
Aug 06 11:41:38 dist-test-slave-45dx krb5kdc[2879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:38.567063 3005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75743732347a4b06bdd4c017310bc74b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37385 } }
I20260806 11:41:38.567325 3005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:38.567399 3005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75743732347a4b06bdd4c017310bc74b, State: Initialized, Role: FOLLOWER
I20260806 11:41:38.567919 3005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75743732347a4b06bdd4c017310bc74b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37385 } }
I20260806 11:41:38.568081 3005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:38.568150 3005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:38.568274 3005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:38.569458 3005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75743732347a4b06bdd4c017310bc74b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37385 } }
I20260806 11:41:38.569828 3005 leader_election.cc:302] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75743732347a4b06bdd4c017310bc74b; no voters:
I20260806 11:41:38.570290 3005 leader_election.cc:288] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:38.570406 3017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:38.570619 3017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [term 1 LEADER]: Becoming Leader. State: Replica: 75743732347a4b06bdd4c017310bc74b, State: Running, Role: LEADER
I20260806 11:41:38.570923 3017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75743732347a4b06bdd4c017310bc74b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37385 } }
I20260806 11:41:38.571549 3005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:38.572558 3018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75743732347a4b06bdd4c017310bc74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75743732347a4b06bdd4c017310bc74b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37385 } } }
I20260806 11:41:38.572685 3018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:38.572986 3022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75743732347a4b06bdd4c017310bc74b. Latest consensus state: current_term: 1 leader_uuid: "75743732347a4b06bdd4c017310bc74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75743732347a4b06bdd4c017310bc74b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37385 } } }
I20260806 11:41:38.573083 3022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:38.573699 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:38.558849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.0.113.62:37385)
0806 11:41:38.559261 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:38.559280 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:38.559320 (+ 40us) server_negotiation.cc:407] Connection header received
0806 11:41:38.560144 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:38.560179 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:38.560634 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:38.560961 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:38.561756 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.563204 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.564074 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.564921 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.565139 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:38.567544 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:38.567550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:38.567565 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:38.567594 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:38.569759 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:38.570325 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:38.570333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:38.570340 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:38.570493 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:38.570824 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:38.570827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:38.570829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:38.571290 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:38.571473 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:38.572254 (+ 781us) server_negotiation.cc:299] Negotiation successful
0806 11:41:38.572619 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":101,"threads_started":1}
W20260806 11:41:38.576740 3039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:41:38.576834 3039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:41:38.576927 3029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:38.577616 3029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:38.583006 3029 catalog_manager.cc:1347] Generated new cluster ID: 8275729aea5043c4ad207acaaf004586
I20260806 11:41:38.583066 3029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:38.591281 3029 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:38.592142 3029 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:38.602391 3029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: Generated new TSK 0
I20260806 11:41:38.603035 3029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:41:38.634931 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37385
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:38.746331 3049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:38.746595 3049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:38.746642 3049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:38.750751 3049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:38.750806 3049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:38.750845 3049 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 11:41:38.750887 3049 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 11:41:38.750964 3049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:38.756201 3049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.113.62:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:38.757237 3049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:38.758263 3049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:38.767889 3055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:38.767990 3049 server_base.cc:1034] running on GCE node
W20260806 11:41:38.767925 3060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:38.767948 3056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:38.768568 3049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:38.769131 3049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:38.770287 3049 hybrid_clock.cc:648] HybridClock initialized: now 1786016498770264 us; error 32 us; skew 500 ppm
Aug 06 11:41:38 dist-test-slave-45dx krb5kdc[2879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:38.773177 3049 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:41:38.774144 3049 webserver.cc:460] Webserver started at http://127.0.113.1:40817/ using document root <none> and password file <none>
I20260806 11:41:38.774691 3049 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:38.774752 3049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:38.774981 3049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:38.777014 3049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "99be8fe64fae48da919741cda24b0b84"
format_stamp: "Formatted at 2026-08-06 11:41:38 on dist-test-slave-45dx"
server_key: "c367a5501d79a305924f8242879b7a2d"
server_key_iv: "e751af7d1b2337525d31f5ca93d3c6b3"
server_key_version: "encryptionkey@0"
I20260806 11:41:38.777570 3049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "99be8fe64fae48da919741cda24b0b84"
format_stamp: "Formatted at 2026-08-06 11:41:38 on dist-test-slave-45dx"
server_key: "c367a5501d79a305924f8242879b7a2d"
server_key_iv: "e751af7d1b2337525d31f5ca93d3c6b3"
server_key_version: "encryptionkey@0"
I20260806 11:41:38.781177 3049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 11:41:38.783430 3068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:38.784610 3049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:41:38.784730 3049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "99be8fe64fae48da919741cda24b0b84"
format_stamp: "Formatted at 2026-08-06 11:41:38 on dist-test-slave-45dx"
server_key: "c367a5501d79a305924f8242879b7a2d"
server_key_iv: "e751af7d1b2337525d31f5ca93d3c6b3"
server_key_version: "encryptionkey@0"
I20260806 11:41:38.784835 3049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:38.801296 3049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:38.803915 3049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:38.804103 3049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:38.805490 3049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:38.805557 3049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:38.805624 3049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:38.805671 3049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 11:41:38 dist-test-slave-45dx krb5kdc[2879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:38.816730 3049 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:35727
I20260806 11:41:38.816771 3241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:35727 every 8 connection(s)
I20260806 11:41:38.817790 3049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:38.819010 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:38.806199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:47775 (local address 127.0.113.62:37385)
0806 11:41:38.806349 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:38.806355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:38.807148 (+ 793us) server_negotiation.cc:407] Connection header received
0806 11:41:38.808078 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:38.808084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:38.808149 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:38.808249 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:38.809423 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.809965 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.811270 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.811610 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.811724 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:38.815082 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:38.815085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:38.815088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:38.815118 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:38.816835 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:38.817414 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:38.817417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:38.817419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:38.817499 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:38.817900 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:38.817903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:38.817905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:38.818088 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:38.818199 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:38.818755 (+ 556us) server_negotiation.cc:299] Negotiation successful
0806 11:41:38.818886 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 11:41:38.819777 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:38.806555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37385 (local address 127.0.113.1:47775)
0806 11:41:38.807001 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:38.807047 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:38.807882 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:38.808471 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:38.808481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:38.808849 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:38.809270 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:38.809282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.810095 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.810104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:38.811150 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:38.811161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.811765 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.811770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:38.811939 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:38.812526 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:38.812547 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:38.814915 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:38.816981 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:38.816986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:38.816998 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:38.817315 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:38.817595 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:38.817598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:38.817600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:38.817775 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:38.818200 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:38.818208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:38.818468 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 11:41:38.818690 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 11:41:38.818948 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":129,"threads_started":1}
I20260806 11:41:38.821118 3243 heartbeater.cc:347] Connected to a master server at 127.0.113.62:37385
I20260806 11:41:38.821259 3243 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:38.821396 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 3049
I20260806 11:41:38.821524 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:38.821828 3243 heartbeater.cc:511] Master 127.0.113.62:37385 requested a full tablet report, sending...
I20260806 11:41:38.821867 452 external_mini_cluster.cc:1428] Setting key e94d8f7a3753892fb865a868adb15007
I20260806 11:41:38.823501 2943 ts_manager.cc:194] Registered new tserver with Master: 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727)
I20260806 11:41:38.824937 2943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:47775
I20260806 11:41:38.831696 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:38.823588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:59663 (local address 127.0.113.62:37385)
0806 11:41:38.823680 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:38.823684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:38.824594 (+ 910us) server_negotiation.cc:407] Connection header received
0806 11:41:38.824680 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:38.824685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:38.824737 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:38.824819 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:38.825322 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.826110 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.827037 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.827257 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.827359 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:38.828547 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:38.828550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:38.828551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:38.828573 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:38.830086 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:38.830584 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:38.830587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:38.830588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:38.830658 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:38.830970 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:38.830972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:38.830973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:38.831121 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:38.831195 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:38.831504 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 11:41:38.831591 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 11:41:38.831844 3249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:38.823530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37385 (local address 127.0.113.1:59663)
0806 11:41:38.824500 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:38.824515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:38.824606 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:38.824854 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:38.824857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:38.825076 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:38.825190 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:38.825197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.826281 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.826286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:38.826910 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:38.826919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:38.827365 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:38.827369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:38.827460 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:38.827909 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:38.827928 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:38.828417 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:38.830194 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:38.830200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:38.830205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:38.830486 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:38.830749 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:38.830754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:38.830758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:38.830868 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:38.831217 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:38.831222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:38.831334 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 11:41:38.831461 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 11:41:38.831610 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":17,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:41:38.851438 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37385
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:38.962510 3259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:38.962760 3259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:38.962805 3259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:38.966732 3259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:38.966796 3259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:38.966836 3259 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 11:41:38.966850 3259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 11:41:38.966888 3259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:41:38.972008 3259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.113.62:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:38.972999 3259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:38.973975 3259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:38.983377 3266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:38.983345 3268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:38.983494 3259 server_base.cc:1034] running on GCE node
W20260806 11:41:38.983359 3265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:38.984027 3259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:38.984709 3259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:38.985889 3259 hybrid_clock.cc:648] HybridClock initialized: now 1786016498985860 us; error 44 us; skew 500 ppm
Aug 06 11:41:38 dist-test-slave-45dx krb5kdc[2879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:38.988795 3259 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:41:38.989660 3259 webserver.cc:460] Webserver started at http://127.0.113.2:41929/ using document root <none> and password file <none>
I20260806 11:41:38.990201 3259 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:38.990262 3259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:38.990502 3259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:38.992503 3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "416c3182b1434e20a2419f908049bb08"
format_stamp: "Formatted at 2026-08-06 11:41:38 on dist-test-slave-45dx"
server_key: "a375be8410b42349037388e6bf63e292"
server_key_iv: "84aa993c58ed1b7e712bff17ce3011de"
server_key_version: "encryptionkey@0"
I20260806 11:41:38.993083 3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "416c3182b1434e20a2419f908049bb08"
format_stamp: "Formatted at 2026-08-06 11:41:38 on dist-test-slave-45dx"
server_key: "a375be8410b42349037388e6bf63e292"
server_key_iv: "84aa993c58ed1b7e712bff17ce3011de"
server_key_version: "encryptionkey@0"
I20260806 11:41:38.996680 3259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:41:38.998991 3278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:39.000093 3259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:39.000224 3259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "416c3182b1434e20a2419f908049bb08"
format_stamp: "Formatted at 2026-08-06 11:41:38 on dist-test-slave-45dx"
server_key: "a375be8410b42349037388e6bf63e292"
server_key_iv: "84aa993c58ed1b7e712bff17ce3011de"
server_key_version: "encryptionkey@0"
I20260806 11:41:39.000350 3259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:39.017289 3259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:39.020118 3259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:39.020269 3259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:39.021787 3259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:39.021840 3259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:39.021901 3259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:39.021934 3259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 11:41:39 dist-test-slave-45dx krb5kdc[2879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:39.033000 3259 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:38359
I20260806 11:41:39.033078 3437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:38359 every 8 connection(s)
I20260806 11:41:39.033875 3259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:41:39.035508 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.022574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:54497 (local address 127.0.113.62:37385)
0806 11:41:39.022737 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.022744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.023595 (+ 851us) server_negotiation.cc:407] Connection header received
0806 11:41:39.024722 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.024727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.024786 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.024845 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:39.026225 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.026804 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.028071 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.028344 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.028466 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.031705 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:39.031709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:39.031716 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:39.031741 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:39.033433 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.033939 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.033943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.033945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.034037 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.034446 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.034449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.034450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.034679 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:39.034783 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.035281 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.035391 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 11:41:39.036281 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.022988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37385 (local address 127.0.113.2:54497)
0806 11:41:39.023430 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.023480 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.024473 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.025252 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.025258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.025569 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:39.026026 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.026039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.026942 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.026946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.027951 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.027961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.028508 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.028512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.028712 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.029337 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:39.029361 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:39.031541 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:39.033547 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.033553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.033566 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.033825 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.034142 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.034144 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.034146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.034326 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.034794 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:39.034800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:39.035068 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.035229 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.035472 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":127,"threads_started":1}
I20260806 11:41:39.037806 3438 heartbeater.cc:347] Connected to a master server at 127.0.113.62:37385
I20260806 11:41:39.037921 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 3259
I20260806 11:41:39.037947 3438 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:39.038043 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:41:39.038311 452 external_mini_cluster.cc:1428] Setting key 895f94ae3a9e09632959a2cc9549c8b8
I20260806 11:41:39.038514 3438 heartbeater.cc:511] Master 127.0.113.62:37385 requested a full tablet report, sending...
I20260806 11:41:39.039636 2943 ts_manager.cc:194] Registered new tserver with Master: 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359)
I20260806 11:41:39.040426 2943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:54497
I20260806 11:41:39.049491 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.040161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:48559 (local address 127.0.113.62:37385)
0806 11:41:39.040291 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.040316 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.040574 (+ 258us) server_negotiation.cc:407] Connection header received
0806 11:41:39.041843 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.041847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.041901 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.041977 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:39.042536 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.043017 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.044283 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.044555 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.044639 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.046012 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:39.046015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:39.046017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:39.046037 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:39.047488 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.047990 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.047992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.047993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.048066 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.048425 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.048431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.048435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.048780 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:39.048890 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.049226 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.049333 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:41:39.049499 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.040072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37385 (local address 127.0.113.2:48559)
0806 11:41:39.040490 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.040506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.040645 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.042080 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.042083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.042292 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:39.042399 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.042407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.043177 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.043183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.044159 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.044175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.044682 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.044689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.044828 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.045377 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:39.045397 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:39.045869 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:39.047607 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.047612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.047616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.047890 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.048181 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.048184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.048186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.048301 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.048940 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:39.048945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:39.049044 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.049149 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.049295 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:41:39.068830 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37385
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:39.182448 3453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:39.182715 3453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:39.182761 3453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:39.186780 3453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:39.186836 3453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:39.186874 3453 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 11:41:39.186889 3453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 11:41:39.186920 3453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:41:39.192096 3453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.113.62:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:39.193109 3453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:39.194068 3453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:39.203761 3459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:39.203751 3458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:39.203751 3461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:39.204088 3453 server_base.cc:1034] running on GCE node
I20260806 11:41:39.204545 3453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:39.205163 3453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:39.206337 3453 hybrid_clock.cc:648] HybridClock initialized: now 1786016499206308 us; error 48 us; skew 500 ppm
Aug 06 11:41:39 dist-test-slave-45dx krb5kdc[2879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:39.209211 3453 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:41:39.210088 3453 webserver.cc:460] Webserver started at http://127.0.113.3:32807/ using document root <none> and password file <none>
I20260806 11:41:39.210638 3453 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:39.210729 3453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:39.211006 3453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:39.213333 3453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "846eea04fb754f439ea1da73c0ffa2ec"
format_stamp: "Formatted at 2026-08-06 11:41:39 on dist-test-slave-45dx"
server_key: "6f9e4be80b91d1733cea818a397a66ca"
server_key_iv: "808a3be5ebe221127b45a47a918f9e31"
server_key_version: "encryptionkey@0"
I20260806 11:41:39.213850 3453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "846eea04fb754f439ea1da73c0ffa2ec"
format_stamp: "Formatted at 2026-08-06 11:41:39 on dist-test-slave-45dx"
server_key: "6f9e4be80b91d1733cea818a397a66ca"
server_key_iv: "808a3be5ebe221127b45a47a918f9e31"
server_key_version: "encryptionkey@0"
I20260806 11:41:39.217466 3453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:41:39.219869 3469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:39.221544 3453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 11:41:39.221724 3453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "846eea04fb754f439ea1da73c0ffa2ec"
format_stamp: "Formatted at 2026-08-06 11:41:39 on dist-test-slave-45dx"
server_key: "6f9e4be80b91d1733cea818a397a66ca"
server_key_iv: "808a3be5ebe221127b45a47a918f9e31"
server_key_version: "encryptionkey@0"
I20260806 11:41:39.221873 3453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:39.242901 3453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:39.246205 3453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:39.246402 3453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:39.247799 3453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:39.247850 3453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:39.247890 3453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:39.247941 3453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 11:41:39 dist-test-slave-45dx krb5kdc[2879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:39.258702 3453 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:41533
I20260806 11:41:39.258764 3635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:41533 every 8 connection(s)
I20260806 11:41:39.259793 3453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:41:39.262262 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.248621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:36349 (local address 127.0.113.62:37385)
0806 11:41:39.248785 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.248791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.249592 (+ 801us) server_negotiation.cc:407] Connection header received
0806 11:41:39.250564 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.250567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.250625 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.250686 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:39.252576 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.253487 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.255302 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.255664 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.255787 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.258701 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:39.258705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:39.258707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:39.258735 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:39.260185 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.260823 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.260826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.260827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.260916 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.261316 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.261318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.261320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.261500 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:39.261579 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.262042 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.262141 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:41:39.263046 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.248996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37385 (local address 127.0.113.3:36349)
0806 11:41:39.249433 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.249481 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.250306 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.251148 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.251155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.251657 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:39.252399 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.252421 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.253633 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.253640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.255144 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.255155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.255843 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.255846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.255991 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.256612 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:39.256636 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:39.258533 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:39.260363 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.260368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.260382 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.260705 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.261046 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.261049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.261054 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.261223 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.261587 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:39.261592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:39.261808 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.261975 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.262206 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":115,"threads_started":1}
I20260806 11:41:39.264220 3637 heartbeater.cc:347] Connected to a master server at 127.0.113.62:37385
I20260806 11:41:39.264422 3637 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:39.264897 3637 heartbeater.cc:511] Master 127.0.113.62:37385 requested a full tablet report, sending...
I20260806 11:41:39.265897 2944 ts_manager.cc:194] Registered new tserver with Master: 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533)
I20260806 11:41:39.266484 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 3453
I20260806 11:41:39.266613 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:41:39.266631 2944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:36349
I20260806 11:41:39.266964 452 external_mini_cluster.cc:1428] Setting key 45b461c221bbfb5916c0aba013504ce0
I20260806 11:41:39.269044 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:41:39.273998 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.266516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37385 (local address 127.0.113.3:54353)
0806 11:41:39.266861 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.266878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.267030 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.267283 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.267286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.267491 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:39.267593 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.267600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.268529 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.268532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.269231 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.269240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.269764 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.269766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.269829 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.270305 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:39.270316 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:39.270613 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:39.272344 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.272348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.272350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.272728 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.273057 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.273060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.273061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.273138 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.273567 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:39.273569 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:39.273661 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.273755 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.273855 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260806 11:41:39.274215 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.266472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:54353 (local address 127.0.113.62:37385)
0806 11:41:39.266575 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.266579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.266964 (+ 385us) server_negotiation.cc:407] Connection header received
0806 11:41:39.267126 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.267129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.267174 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.267270 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:39.267696 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.268401 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.269333 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.269656 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.269775 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.270745 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:39.270751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:39.270757 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:39.270787 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:39.272197 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.272836 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.272842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.272846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.272958 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.273229 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.273234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.273239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.273469 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:39.273568 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.273863 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.273995 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 11:41:39 dist-test-slave-45dx krb5kdc[2879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 11:41:39 dist-test-slave-45dx krb5kdc[2879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:39.296486 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.288288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.0.113.62:37385)
0806 11:41:39.288487 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.288491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.288505 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:41:39.288631 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.288634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.288676 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.288757 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:39.289440 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.289941 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.290771 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.290987 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.291096 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.293462 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:39.293470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:39.293472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:39.293494 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:39.294861 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.295280 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.295284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.295286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.295363 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.295632 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.295636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.295637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.295803 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:39.295958 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.296185 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.296294 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 11:41:39.299109 2944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:41:39.302194 2944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:39.316242 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.310430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:35727 (local address 127.0.0.1:58038)
0806 11:41:39.311005 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.311056 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.311186 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.312072 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.312078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.312100 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:39.312204 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.312208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.313400 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.313403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.314462 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.314471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.315722 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.315725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.315828 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.315861 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.315974 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.316094 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"spinlock_wait_cycles":1792,"thread_start_us":81,"threads_started":1}
I20260806 11:41:39.316293 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.310607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.0.113.1:35727)
0806 11:41:39.311515 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.311533 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.311558 (+ 25us) server_negotiation.cc:407] Connection header received
0806 11:41:39.311640 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.311649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.311898 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.312076 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:39.312396 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.313279 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.314605 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.315400 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.315479 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.315628 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.316035 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.316137 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":84,"threads_started":1}
I20260806 11:41:39.317406 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.310628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41533 (local address 127.0.0.1:44640)
0806 11:41:39.311007 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.311057 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.311186 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.312808 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.312811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.312830 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:39.312932 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.312938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.314597 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.314600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.315615 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.315626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.316964 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.316968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.317068 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.317099 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.317170 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.317236 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"spinlock_wait_cycles":768,"thread_start_us":162,"threads_started":1}
I20260806 11:41:39.317802 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.311556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.0.113.3:41533)
0806 11:41:39.312435 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.312454 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.312474 (+ 20us) server_negotiation.cc:407] Connection header received
0806 11:41:39.312536 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.312540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.312681 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.312796 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:39.313058 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.314462 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.315766 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.316807 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.316934 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.317093 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.317592 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.317671 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":129,"threads_started":1}
I20260806 11:41:39.317802 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.311595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.0.113.2:38359)
0806 11:41:39.312012 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.312025 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.312294 (+ 269us) server_negotiation.cc:407] Connection header received
0806 11:41:39.312962 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.312966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.313173 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.313335 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:39.313620 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.314773 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.316144 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.317061 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.317171 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.317317 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.317570 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.317662 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":121,"threads_started":1}
I20260806 11:41:39.317857 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.311151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:38359 (local address 127.0.0.1:49850)
0806 11:41:39.312172 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.312189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.312319 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.313321 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.313327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.313352 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:39.313500 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.313507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.314891 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.314895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.316017 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.316026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.317187 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.317191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.317310 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.317335 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.317418 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.317722 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":37,"thread_start_us":113,"threads_started":1}
I20260806 11:41:39.319696 3136 tablet_service.cc:1467] Processing CreateTablet for tablet 54f0c2658db34ae1b6570b91f48c82e5 (DEFAULT_TABLE table=test-table [id=ad4d9d7c4c4e4322af4b1371be3190cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:39.320468 3543 tablet_service.cc:1467] Processing CreateTablet for tablet 54f0c2658db34ae1b6570b91f48c82e5 (DEFAULT_TABLE table=test-table [id=ad4d9d7c4c4e4322af4b1371be3190cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:39.320958 3136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f0c2658db34ae1b6570b91f48c82e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:39.321444 3543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f0c2658db34ae1b6570b91f48c82e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:39.321141 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 54f0c2658db34ae1b6570b91f48c82e5 (DEFAULT_TABLE table=test-table [id=ad4d9d7c4c4e4322af4b1371be3190cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:39.322093 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f0c2658db34ae1b6570b91f48c82e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:39.327739 3671 tablet_bootstrap.cc:492] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: Bootstrap starting.
I20260806 11:41:39.329634 3672 tablet_bootstrap.cc:492] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: Bootstrap starting.
I20260806 11:41:39.331705 3671 tablet_bootstrap.cc:654] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:39.331944 3672 tablet_bootstrap.cc:654] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:39.332593 3673 tablet_bootstrap.cc:492] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: Bootstrap starting.
I20260806 11:41:39.332645 3671 log.cc:824] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:39.332690 3672 log.cc:824] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:39.334700 3673 tablet_bootstrap.cc:654] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:39.334805 3671 tablet_bootstrap.cc:492] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: No bootstrap required, opened a new log
I20260806 11:41:39.335072 3671 ts_tablet_manager.cc:1397] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 11:41:39.335563 3673 log.cc:824] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:39.336410 3672 tablet_bootstrap.cc:492] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: No bootstrap required, opened a new log
I20260806 11:41:39.336606 3672 ts_tablet_manager.cc:1397] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260806 11:41:39.337303 3673 tablet_bootstrap.cc:492] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: No bootstrap required, opened a new log
I20260806 11:41:39.337495 3673 ts_tablet_manager.cc:1397] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:41:39.339330 3671 raft_consensus.cc:348] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } }
I20260806 11:41:39.339645 3671 raft_consensus.cc:374] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:39.339709 3671 raft_consensus.cc:729] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be8fe64fae48da919741cda24b0b84, State: Initialized, Role: FOLLOWER
I20260806 11:41:39.340217 3671 consensus_queue.cc:260] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } }
I20260806 11:41:39.340687 3673 raft_consensus.cc:348] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } }
I20260806 11:41:39.340916 3243 heartbeater.cc:503] Master 127.0.113.62:37385 was elected leader, sending a full tablet report...
I20260806 11:41:39.340955 3673 raft_consensus.cc:374] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:39.340814 3672 raft_consensus.cc:348] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } }
I20260806 11:41:39.341032 3673 raft_consensus.cc:729] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416c3182b1434e20a2419f908049bb08, State: Initialized, Role: FOLLOWER
I20260806 11:41:39.341059 3672 raft_consensus.cc:374] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:39.341127 3672 raft_consensus.cc:729] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846eea04fb754f439ea1da73c0ffa2ec, State: Initialized, Role: FOLLOWER
I20260806 11:41:39.341459 3673 consensus_queue.cc:260] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } }
I20260806 11:41:39.341571 3672 consensus_queue.cc:260] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } }
I20260806 11:41:39.342483 3673 ts_tablet_manager.cc:1428] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 11:41:39.342612 3672 ts_tablet_manager.cc:1428] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 11:41:39.342955 3637 heartbeater.cc:503] Master 127.0.113.62:37385 was elected leader, sending a full tablet report...
I20260806 11:41:39.342972 3671 ts_tablet_manager.cc:1428] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 11:41:39.343293 3438 heartbeater.cc:503] Master 127.0.113.62:37385 was elected leader, sending a full tablet report...
I20260806 11:41:39.346489 3678 raft_consensus.cc:482] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:39.346614 3678 raft_consensus.cc:504] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } }
I20260806 11:41:39.347601 3678 leader_election.cc:288] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727), 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533)
I20260806 11:41:39.351509 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.347791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:35727 (local address 127.0.113.2:34851)
0806 11:41:39.347916 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.347935 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.348028 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.348395 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.348397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.348417 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:39.348549 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.348556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.349403 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.349408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.350252 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.350280 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.351162 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.351165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.351239 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.351259 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.351337 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.351390 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 11:41:39.351588 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.347825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:34851 (local address 127.0.113.1:35727)
0806 11:41:39.347938 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.347942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.347995 (+ 53us) server_negotiation.cc:407] Connection header received
0806 11:41:39.348138 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.348142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.348202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.348283 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:39.348664 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.349287 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.350404 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.351051 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.351161 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.351219 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.351378 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.351447 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 11:41:39.352202 3164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f0c2658db34ae1b6570b91f48c82e5" candidate_uuid: "416c3182b1434e20a2419f908049bb08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be8fe64fae48da919741cda24b0b84" is_pre_election: true
I20260806 11:41:39.352523 3164 raft_consensus.cc:2454] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416c3182b1434e20a2419f908049bb08 in term 0.
I20260806 11:41:39.352713 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.347843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41533 (local address 127.0.113.2:41251)
0806 11:41:39.348152 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.348166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.348260 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.348529 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.348532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.348547 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:39.348680 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.348688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.349477 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.349479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.351408 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.351418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.352250 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.352254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.352385 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.352409 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.352498 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.352577 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
I20260806 11:41:39.352746 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.348191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:41251 (local address 127.0.113.3:41533)
0806 11:41:39.348295 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.348337 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.348354 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:41:39.348394 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.348397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.348436 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.348511 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:39.348786 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.349360 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.351536 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.352107 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.352188 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.352229 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.352539 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.352601 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 11:41:39.353261 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f0c2658db34ae1b6570b91f48c82e5" candidate_uuid: "416c3182b1434e20a2419f908049bb08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846eea04fb754f439ea1da73c0ffa2ec" is_pre_election: true
I20260806 11:41:39.353278 3280 leader_election.cc:302] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416c3182b1434e20a2419f908049bb08, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:39.353489 3569 raft_consensus.cc:2454] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416c3182b1434e20a2419f908049bb08 in term 0.
I20260806 11:41:39.353523 3678 raft_consensus.cc:2781] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:39.353600 3678 raft_consensus.cc:482] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:39.353655 3678 raft_consensus.cc:3037] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:39.354725 3678 raft_consensus.cc:504] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } }
I20260806 11:41:39.355077 3678 leader_election.cc:288] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [CANDIDATE]: Term 1 election: Requested vote from peers 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727), 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533)
I20260806 11:41:39.355396 3164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f0c2658db34ae1b6570b91f48c82e5" candidate_uuid: "416c3182b1434e20a2419f908049bb08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be8fe64fae48da919741cda24b0b84"
I20260806 11:41:39.355456 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f0c2658db34ae1b6570b91f48c82e5" candidate_uuid: "416c3182b1434e20a2419f908049bb08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846eea04fb754f439ea1da73c0ffa2ec"
I20260806 11:41:39.355505 3164 raft_consensus.cc:3037] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:39.355583 3569 raft_consensus.cc:3037] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:39.356647 3164 raft_consensus.cc:2454] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416c3182b1434e20a2419f908049bb08 in term 1.
I20260806 11:41:39.356657 3569 raft_consensus.cc:2454] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416c3182b1434e20a2419f908049bb08 in term 1.
I20260806 11:41:39.356933 3280 leader_election.cc:302] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416c3182b1434e20a2419f908049bb08, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:39.357228 3678 raft_consensus.cc:2781] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:39.357445 3678 raft_consensus.cc:686] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 1 LEADER]: Becoming Leader. State: Replica: 416c3182b1434e20a2419f908049bb08, State: Running, Role: LEADER
I20260806 11:41:39.357750 3678 consensus_queue.cc:237] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } }
I20260806 11:41:39.360901 2943 catalog_manager.cc:5495] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 416c3182b1434e20a2419f908049bb08 (127.0.113.2). New cstate: current_term: 1 leader_uuid: "416c3182b1434e20a2419f908049bb08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } health_report { overall_health: HEALTHY } } }
Aug 06 11:41:39 dist-test-slave-45dx krb5kdc[2879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:39.406728 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.397918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.0.113.62:37385)
0806 11:41:39.398089 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.398096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.398354 (+ 258us) server_negotiation.cc:407] Connection header received
0806 11:41:39.398440 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.398442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.398482 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.398566 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:39.399259 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.399767 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.400767 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.401005 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.401086 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.403376 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:39.403379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:39.403381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:39.403402 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:39.404950 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.405416 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.405421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.405423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.405500 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:39.405781 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:39.405784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:39.405786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:39.405984 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:39.406090 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.406421 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.406556 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":74}
I20260806 11:41:39.406728 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.397879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37385 (local address 127.0.0.1:54534)
0806 11:41:39.398257 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.398270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.398369 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.398623 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.398626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.399044 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:39.399141 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.399147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.399913 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.399917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.400658 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.400667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.401109 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.401112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.401172 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.401743 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:39.401754 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:39.403221 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:39.405058 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.405070 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.405076 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.405318 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.405594 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:39.405597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:39.405599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:39.405683 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:39.406103 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:39.406112 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:39.406220 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.406339 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.406502 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":109,"threads_started":1}
I20260806 11:41:39.410539 2943 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:54534:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 11:41:39.411228 2943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:39.415859 3136 tablet_service.cc:1467] Processing CreateTablet for tablet 72c4ab641ffe41c3988db0372571b7d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d849d4511df045cfa4040f9be24fffaa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 11:41:39.415877 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 72c4ab641ffe41c3988db0372571b7d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d849d4511df045cfa4040f9be24fffaa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 11:41:39.416083 3543 tablet_service.cc:1467] Processing CreateTablet for tablet 72c4ab641ffe41c3988db0372571b7d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d849d4511df045cfa4040f9be24fffaa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 11:41:39.416635 3136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c4ab641ffe41c3988db0372571b7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:39.416653 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c4ab641ffe41c3988db0372571b7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:39.416653 3543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c4ab641ffe41c3988db0372571b7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:39.419234 3671 tablet_bootstrap.cc:492] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: Bootstrap starting.
I20260806 11:41:39.419488 3673 tablet_bootstrap.cc:492] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: Bootstrap starting.
I20260806 11:41:39.420203 3671 tablet_bootstrap.cc:654] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:39.420413 3673 tablet_bootstrap.cc:654] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:39.420472 3672 tablet_bootstrap.cc:492] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: Bootstrap starting.
I20260806 11:41:39.421626 3672 tablet_bootstrap.cc:654] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:39.421952 3673 tablet_bootstrap.cc:492] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: No bootstrap required, opened a new log
I20260806 11:41:39.422055 3673 ts_tablet_manager.cc:1397] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 11:41:39.422580 3673 raft_consensus.cc:348] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.422706 3673 raft_consensus.cc:374] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:39.422751 3673 raft_consensus.cc:729] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416c3182b1434e20a2419f908049bb08, State: Initialized, Role: FOLLOWER
I20260806 11:41:39.422858 3672 tablet_bootstrap.cc:492] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: No bootstrap required, opened a new log
I20260806 11:41:39.422888 3673 consensus_queue.cc:260] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.422971 3672 ts_tablet_manager.cc:1397] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 11:41:39.423220 3673 ts_tablet_manager.cc:1428] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260806 11:41:39.423215 3678 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.423359 3678 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:39.423429 3672 raft_consensus.cc:348] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.423542 3672 raft_consensus.cc:374] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:39.423592 3672 raft_consensus.cc:729] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846eea04fb754f439ea1da73c0ffa2ec, State: Initialized, Role: FOLLOWER
I20260806 11:41:39.423731 3672 consensus_queue.cc:260] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.424026 3672 ts_tablet_manager.cc:1428] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 11:41:39.423975 3679 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.424104 3679 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:39.424809 3671 tablet_bootstrap.cc:492] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: No bootstrap required, opened a new log
I20260806 11:41:39.424911 3671 ts_tablet_manager.cc:1397] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 11:41:39.425328 3671 raft_consensus.cc:348] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.425447 3671 raft_consensus.cc:374] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:39.425489 3671 raft_consensus.cc:729] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be8fe64fae48da919741cda24b0b84, State: Initialized, Role: FOLLOWER
I20260806 11:41:39.425614 3671 consensus_queue.cc:260] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.425887 3671 ts_tablet_manager.cc:1428] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 11:41:39.425921 3677 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.426069 3677 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 11:41:39.510854 3638 tablet.cc:2369] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:41:39.535024 3440 tablet.cc:2369] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:41:39.569408 3244 tablet.cc:2369] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:41:39.776893 3679 raft_consensus.cc:482] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:39.777050 3679 raft_consensus.cc:504] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.777576 3677 raft_consensus.cc:482] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:39.777709 3677 raft_consensus.cc:504] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.778271 3679 leader_election.cc:288] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727), 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359)
I20260806 11:41:39.778816 3677 leader_election.cc:288] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359), 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533)
I20260806 11:41:39.783794 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.778463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:35727 (local address 127.0.113.3:38551)
0806 11:41:39.778714 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.778736 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.778848 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.779772 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.779778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.779801 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:39.779916 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.779924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.780951 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.780954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.782333 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.782350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.783326 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.783331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.783435 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.783494 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.783645 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.783695 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":129,"thread_start_us":105,"threads_started":1}
I20260806 11:41:39.783823 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.778593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:38551 (local address 127.0.113.1:35727)
0806 11:41:39.778718 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.778728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.779369 (+ 641us) server_negotiation.cc:407] Connection header received
0806 11:41:39.779437 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.779441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.779491 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.779568 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:39.780055 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.780806 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.782479 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.783083 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.783160 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.783235 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.783609 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.783696 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":50}
I20260806 11:41:39.784025 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.779132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:45055 (local address 127.0.113.2:38359)
0806 11:41:39.779345 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.779349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.779365 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:41:39.779421 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.779427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.779480 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.779565 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:39.779838 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.780699 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.782479 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.783143 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.783225 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.783293 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.783841 (+ 548us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.783913 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":124}
I20260806 11:41:39.784327 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.779208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:57041 (local address 127.0.113.2:38359)
0806 11:41:39.779475 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.779479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.780007 (+ 528us) server_negotiation.cc:407] Connection header received
0806 11:41:39.780172 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.780174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.780219 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.780304 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:39.780626 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.781377 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.782530 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.783201 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.783294 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.783347 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.783755 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.783820 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":202,"thread_start_us":70,"threads_started":1}
I20260806 11:41:39.784523 3164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c4ab641ffe41c3988db0372571b7d1" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be8fe64fae48da919741cda24b0b84" is_pre_election: true
I20260806 11:41:39.784684 3164 raft_consensus.cc:2454] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 0.
I20260806 11:41:39.784698 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.779045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:38359 (local address 127.0.113.1:57041)
0806 11:41:39.779932 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.779945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.780073 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.780332 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.780335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.780352 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:39.780499 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.780509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.781494 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.781498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.782339 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.782369 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.783440 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.783445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.783543 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.783565 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.783628 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.783696 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":84,"threads_started":1}
I20260806 11:41:39.783794 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.778856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:38359 (local address 127.0.113.3:45055)
0806 11:41:39.779166 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.779179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.779268 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.779574 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.779577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.779595 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:39.779712 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.779719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.780824 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.780827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.782333 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.782346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.783349 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.783352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.783445 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.783492 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.783563 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.783638 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":139,"thread_start_us":82,"threads_started":1}
I20260806 11:41:39.785074 3475 leader_election.cc:302] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 846eea04fb754f439ea1da73c0ffa2ec, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:39.785318 3379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c4ab641ffe41c3988db0372571b7d1" candidate_uuid: "99be8fe64fae48da919741cda24b0b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08" is_pre_election: true
I20260806 11:41:39.785486 3679 raft_consensus.cc:2781] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:39.785543 3679 raft_consensus.cc:482] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:39.785578 3679 raft_consensus.cc:3037] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:39.785548 3379 raft_consensus.cc:2454] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be8fe64fae48da919741cda24b0b84 in term 0.
I20260806 11:41:39.785900 3378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c4ab641ffe41c3988db0372571b7d1" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08" is_pre_election: true
I20260806 11:41:39.786023 3378 raft_consensus.cc:2454] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 0.
I20260806 11:41:39.786022 3070 leader_election.cc:302] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416c3182b1434e20a2419f908049bb08, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:39.786290 3677 raft_consensus.cc:2781] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:39.786345 3677 raft_consensus.cc:482] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:39.786383 3677 raft_consensus.cc:3037] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:39.786449 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.779386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:37689 (local address 127.0.113.3:41533)
0806 11:41:39.779486 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.779490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.779690 (+ 200us) server_negotiation.cc:407] Connection header received
0806 11:41:39.779819 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.779823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.779874 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.779939 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:39.782722 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.783400 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.785217 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.785804 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.785893 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.785942 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.786276 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.786343 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":768}
I20260806 11:41:39.786933 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.779013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41533 (local address 127.0.113.1:37689)
0806 11:41:39.779613 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.779626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.779718 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.780031 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.780035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.780055 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:39.782592 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.782598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.783524 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.783527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.785071 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.785084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.786036 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.786039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.786129 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.786152 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.786646 (+ 494us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.786724 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":4,"thread_start_us":145,"threads_started":1}
I20260806 11:41:39.786908 3679 raft_consensus.cc:504] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.787312 3679 leader_election.cc:288] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 election: Requested vote from peers 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727), 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359)
I20260806 11:41:39.787540 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c4ab641ffe41c3988db0372571b7d1" candidate_uuid: "99be8fe64fae48da919741cda24b0b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846eea04fb754f439ea1da73c0ffa2ec" is_pre_election: true
I20260806 11:41:39.787654 3164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c4ab641ffe41c3988db0372571b7d1" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be8fe64fae48da919741cda24b0b84"
I20260806 11:41:39.787628 3677 raft_consensus.cc:504] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.787760 3569 raft_consensus.cc:2379] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99be8fe64fae48da919741cda24b0b84 in current term 1: Already voted for candidate 846eea04fb754f439ea1da73c0ffa2ec in this term.
I20260806 11:41:39.787838 3164 raft_consensus.cc:2379] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 846eea04fb754f439ea1da73c0ffa2ec in current term 1: Already voted for candidate 99be8fe64fae48da919741cda24b0b84 in this term.
I20260806 11:41:39.787981 3677 leader_election.cc:288] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [CANDIDATE]: Term 1 election: Requested vote from peers 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359), 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533)
I20260806 11:41:39.788332 3378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c4ab641ffe41c3988db0372571b7d1" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08"
I20260806 11:41:39.788420 3379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c4ab641ffe41c3988db0372571b7d1" candidate_uuid: "99be8fe64fae48da919741cda24b0b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08"
I20260806 11:41:39.788450 3378 raft_consensus.cc:3037] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:39.788625 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c4ab641ffe41c3988db0372571b7d1" candidate_uuid: "99be8fe64fae48da919741cda24b0b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846eea04fb754f439ea1da73c0ffa2ec"
I20260806 11:41:39.788764 3569 raft_consensus.cc:2379] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99be8fe64fae48da919741cda24b0b84 in current term 1: Already voted for candidate 846eea04fb754f439ea1da73c0ffa2ec in this term.
I20260806 11:41:39.789041 3069 leader_election.cc:302] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99be8fe64fae48da919741cda24b0b84; no voters: 416c3182b1434e20a2419f908049bb08, 846eea04fb754f439ea1da73c0ffa2ec
I20260806 11:41:39.789198 3677 raft_consensus.cc:2726] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 11:41:39.789659 3378 raft_consensus.cc:2454] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 1.
I20260806 11:41:39.789987 3475 leader_election.cc:302] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 416c3182b1434e20a2419f908049bb08, 846eea04fb754f439ea1da73c0ffa2ec; no voters: 99be8fe64fae48da919741cda24b0b84
I20260806 11:41:39.790166 3679 raft_consensus.cc:2781] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:39.790337 3679 raft_consensus.cc:686] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 LEADER]: Becoming Leader. State: Replica: 846eea04fb754f439ea1da73c0ffa2ec, State: Running, Role: LEADER
I20260806 11:41:39.790684 3679 consensus_queue.cc:237] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.792960 3708 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: TxnStatusTablet state changed. Reason: New leader 846eea04fb754f439ea1da73c0ffa2ec. Latest consensus state: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.793076 3708 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: This TxnStatusTablet replica's current role is: LEADER
I20260806 11:41:39.793366 3710 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 11:41:39.793620 3710 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 11:41:39.794087 2943 catalog_manager.cc:5495] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3). New cstate: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } health_report { overall_health: HEALTHY } } }
I20260806 11:41:39.822682 3678 consensus_queue.cc:1035] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:39.831193 3684 consensus_queue.cc:1035] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:39.871490 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.867695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.0.113.3:41533)
0806 11:41:39.867829 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.867833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.867884 (+ 51us) server_negotiation.cc:407] Connection header received
0806 11:41:39.868088 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.868090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.868143 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.868225 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:39.868507 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.868998 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.870167 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.870378 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.870465 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.870775 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:39.871145 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:39.871228 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.871326 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.871372 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 11:41:39.871546 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.867579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41533 (local address 127.0.0.1:44642)
0806 11:41:39.867821 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.867837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.867951 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.868263 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.868267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.868275 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:41:39.868418 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.868425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.869099 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.869102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.870053 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.870064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.870490 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.870493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.870595 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.870646 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:41:39.871229 (+ 583us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:41:39.871244 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.871317 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.871405 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260806 11:41:39.876528 3164 raft_consensus.cc:1261] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Refusing update from remote peer 846eea04fb754f439ea1da73c0ffa2ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:39.876729 3378 raft_consensus.cc:1261] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Refusing update from remote peer 846eea04fb754f439ea1da73c0ffa2ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:39.877020 3679 consensus_queue.cc:1035] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:39.877295 3708 consensus_queue.cc:1035] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:39.879870 3677 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: TxnStatusTablet state changed. Reason: New leader 846eea04fb754f439ea1da73c0ffa2ec. Latest consensus state: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.879994 3677 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:39.879942 3684 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: TxnStatusTablet state changed. Reason: New leader 846eea04fb754f439ea1da73c0ffa2ec. Latest consensus state: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.880046 3684 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:39.882529 3716 mvcc.cc:205] Tried to move back new op lower bound from 7315523583488716800 to 7315523583145213952. Current Snapshot: MvccSnapshot[applied={T|T < 7315523583488716800}]
I20260806 11:41:39.882709 3708 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.882807 3708 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: This TxnStatusTablet replica's current role is: LEADER
I20260806 11:41:39.883208 3684 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.883314 3684 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 416c3182b1434e20a2419f908049bb08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:39.883327 3713 mvcc.cc:205] Tried to move back new op lower bound from 7315523583488716800 to 7315523583145213952. Current Snapshot: MvccSnapshot[applied={T|T < 7315523583488716800}]
I20260806 11:41:39.883359 3708 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.883467 3708 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 846eea04fb754f439ea1da73c0ffa2ec: This TxnStatusTablet replica's current role is: LEADER
I20260806 11:41:39.884529 3677 tablet_replica.cc:439] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } } }
I20260806 11:41:39.884637 3677 tablet_replica.cc:442] T 72c4ab641ffe41c3988db0372571b7d1 P 99be8fe64fae48da919741cda24b0b84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:41:39.896859 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.892845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.0.113.2:38359)
0806 11:41:39.892982 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.892987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.893243 (+ 256us) server_negotiation.cc:407] Connection header received
0806 11:41:39.893411 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.893414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.893462 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.893541 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:39.893810 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.894288 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.895504 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.895695 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.895769 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.896068 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:39.896476 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:39.896561 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.896692 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.896736 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:41:39.907385 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.903838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41533 (local address 127.0.113.2:59615)
0806 11:41:39.904213 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.904225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.904384 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.904647 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.904650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.904657 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:41:39.904751 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.904757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.905498 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.905503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.906103 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.906112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.906629 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.906632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.906714 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.906786 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:41:39.907104 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:41:39.907113 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.907184 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.907258 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":116,"threads_started":1}
I20260806 11:41:39.907405 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.903927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:59615 (local address 127.0.113.3:41533)
0806 11:41:39.904055 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.904059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.904318 (+ 259us) server_negotiation.cc:407] Connection header received
0806 11:41:39.904506 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.904509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.904553 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.904625 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:39.904854 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.905391 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.906229 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.906523 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.906626 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.906950 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:39.907023 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:39.907099 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.907210 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.907266 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:41:39.932842 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.929194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:44561 (local address 127.0.113.2:38359)
0806 11:41:39.929304 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:39.929308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:39.929476 (+ 168us) server_negotiation.cc:407] Connection header received
0806 11:41:39.929597 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:39.929600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:39.929642 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:39.929725 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:39.929971 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.930465 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.931481 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.931884 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.932000 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.932450 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:39.932523 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:39.932585 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:39.932701 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 11:41:39.932742 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:41:39.933028 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:39.929129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:38359 (local address 127.0.113.3:44561)
0806 11:41:39.929401 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:39.929418 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:39.929518 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:39.929734 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:39.929737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:39.929744 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:41:39.929866 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.929871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.930571 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.930576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.931349 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:39.931360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:39.932078 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:39.932082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:39.932179 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:39.932228 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:41:39.932614 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:41:39.932622 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 11:41:39.932696 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 11:41:39.932780 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260806 11:41:39.961632 2943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:39.961865 2943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:39.964448 2943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: Sending DeleteTablet for 3 replicas of tablet 54f0c2658db34ae1b6570b91f48c82e5
I20260806 11:41:39.965175 3136 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f0c2658db34ae1b6570b91f48c82e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:39 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:58038
I20260806 11:41:39.965224 3543 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f0c2658db34ae1b6570b91f48c82e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:39 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:44640
I20260806 11:41:39.965426 3351 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f0c2658db34ae1b6570b91f48c82e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:39 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:49850
I20260806 11:41:39.965860 3746 tablet_replica.cc:330] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: stopping tablet replica
I20260806 11:41:39.966197 3746 raft_consensus.cc:2229] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:41:39.966246 3744 tablet_replica.cc:330] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: stopping tablet replica
I20260806 11:41:39.966527 3744 raft_consensus.cc:2229] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:39.966487 2943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 11:41:39.966706 3744 raft_consensus.cc:2258] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 11:41:39.966923 2943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:39.967029 3743 tablet_replica.cc:330] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: stopping tablet replica
I20260806 11:41:39.967303 3743 raft_consensus.cc:2229] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:39.967520 3743 raft_consensus.cc:2258] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:39.967856 3746 raft_consensus.cc:2258] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:39.969170 3744 ts_tablet_manager.cc:1905] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:39.969170 3746 ts_tablet_manager.cc:1905] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:39.972486 3543 tablet_service.cc:1467] Processing CreateTablet for tablet 5f885bbce828414f86d67332b7cd6dbb (DEFAULT_TABLE table=test-table [id=9504fa3ed0c048788f98cdbb770309a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:39.972522 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 5f885bbce828414f86d67332b7cd6dbb (DEFAULT_TABLE table=test-table [id=9504fa3ed0c048788f98cdbb770309a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:39.972622 3746 ts_tablet_manager.cc:1918] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:41:39.972702 3746 log.cc:1196] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/54f0c2658db34ae1b6570b91f48c82e5
I20260806 11:41:39.972764 3543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f885bbce828414f86d67332b7cd6dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:39.972777 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f885bbce828414f86d67332b7cd6dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:39.973064 3746 ts_tablet_manager.cc:1939] T 54f0c2658db34ae1b6570b91f48c82e5 P 416c3182b1434e20a2419f908049bb08: Deleting consensus metadata
I20260806 11:41:39.974294 2923 catalog_manager.cc:4841] TS 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359): tablet 54f0c2658db34ae1b6570b91f48c82e5 (table test-table [id=ad4d9d7c4c4e4322af4b1371be3190cd]) successfully deleted
I20260806 11:41:39.975291 3747 tablet_bootstrap.cc:492] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: Bootstrap starting.
I20260806 11:41:39.975935 3136 tablet_service.cc:1467] Processing CreateTablet for tablet 5f885bbce828414f86d67332b7cd6dbb (DEFAULT_TABLE table=test-table [id=9504fa3ed0c048788f98cdbb770309a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:39.976094 3743 ts_tablet_manager.cc:1905] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:39.976234 3136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f885bbce828414f86d67332b7cd6dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:39.976406 3747 tablet_bootstrap.cc:654] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:39.978495 3750 tablet_bootstrap.cc:492] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: Bootstrap starting.
I20260806 11:41:39.979491 3750 tablet_bootstrap.cc:654] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:39.979592 3743 ts_tablet_manager.cc:1918] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:41:39.979682 3743 log.cc:1196] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/54f0c2658db34ae1b6570b91f48c82e5
I20260806 11:41:39.980006 3743 ts_tablet_manager.cc:1939] T 54f0c2658db34ae1b6570b91f48c82e5 P 99be8fe64fae48da919741cda24b0b84: Deleting consensus metadata
I20260806 11:41:39.980685 3750 tablet_bootstrap.cc:492] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: No bootstrap required, opened a new log
I20260806 11:41:39.980805 3750 ts_tablet_manager.cc:1397] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:39.981280 3750 raft_consensus.cc:348] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.981356 2923 catalog_manager.cc:4841] TS 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727): tablet 54f0c2658db34ae1b6570b91f48c82e5 (table test-table [id=ad4d9d7c4c4e4322af4b1371be3190cd]) successfully deleted
I20260806 11:41:39.981410 3750 raft_consensus.cc:374] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:39.981446 3750 raft_consensus.cc:729] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be8fe64fae48da919741cda24b0b84, State: Initialized, Role: FOLLOWER
I20260806 11:41:39.981542 3744 ts_tablet_manager.cc:1918] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:41:39.981571 3750 consensus_queue.cc:260] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.981642 3744 log.cc:1196] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/54f0c2658db34ae1b6570b91f48c82e5
I20260806 11:41:39.981884 3750 ts_tablet_manager.cc:1428] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:39.982383 3744 ts_tablet_manager.cc:1939] T 54f0c2658db34ae1b6570b91f48c82e5 P 846eea04fb754f439ea1da73c0ffa2ec: Deleting consensus metadata
I20260806 11:41:39.982451 3747 tablet_bootstrap.cc:492] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: No bootstrap required, opened a new log
I20260806 11:41:39.982546 3747 ts_tablet_manager.cc:1397] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260806 11:41:39.983003 3747 raft_consensus.cc:348] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.983121 3747 raft_consensus.cc:374] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:39.983165 3747 raft_consensus.cc:729] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416c3182b1434e20a2419f908049bb08, State: Initialized, Role: FOLLOWER
I20260806 11:41:39.983311 3747 consensus_queue.cc:260] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.983570 3748 tablet_bootstrap.cc:492] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: Bootstrap starting.
I20260806 11:41:39.983608 3747 ts_tablet_manager.cc:1428] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 11:41:39.983589 2921 catalog_manager.cc:4841] TS 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533): tablet 54f0c2658db34ae1b6570b91f48c82e5 (table test-table [id=ad4d9d7c4c4e4322af4b1371be3190cd]) successfully deleted
I20260806 11:41:39.984422 3748 tablet_bootstrap.cc:654] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:39.985966 3748 tablet_bootstrap.cc:492] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: No bootstrap required, opened a new log
I20260806 11:41:39.986059 3748 ts_tablet_manager.cc:1397] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 11:41:39.986511 3748 raft_consensus.cc:348] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.986636 3748 raft_consensus.cc:374] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:39.986672 3748 raft_consensus.cc:729] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846eea04fb754f439ea1da73c0ffa2ec, State: Initialized, Role: FOLLOWER
I20260806 11:41:39.986814 3748 consensus_queue.cc:260] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:39.987159 3748 ts_tablet_manager.cc:1428] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 11:41:40.059892 3677 raft_consensus.cc:482] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:40.060065 3677 raft_consensus.cc:504] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:40.060559 3677 leader_election.cc:288] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359), 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533)
I20260806 11:41:40.061113 3378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f885bbce828414f86d67332b7cd6dbb" candidate_uuid: "99be8fe64fae48da919741cda24b0b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08" is_pre_election: true
I20260806 11:41:40.061290 3378 raft_consensus.cc:2454] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be8fe64fae48da919741cda24b0b84 in term 0.
I20260806 11:41:40.061332 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f885bbce828414f86d67332b7cd6dbb" candidate_uuid: "99be8fe64fae48da919741cda24b0b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846eea04fb754f439ea1da73c0ffa2ec" is_pre_election: true
I20260806 11:41:40.061491 3569 raft_consensus.cc:2454] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be8fe64fae48da919741cda24b0b84 in term 0.
I20260806 11:41:40.061657 3070 leader_election.cc:302] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416c3182b1434e20a2419f908049bb08, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:40.061868 3677 raft_consensus.cc:2781] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:40.061945 3677 raft_consensus.cc:482] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:40.061995 3677 raft_consensus.cc:3037] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:40.063910 3677 raft_consensus.cc:504] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:40.064267 3677 leader_election.cc:288] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [CANDIDATE]: Term 1 election: Requested vote from peers 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359), 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533)
I20260806 11:41:40.064646 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f885bbce828414f86d67332b7cd6dbb" candidate_uuid: "99be8fe64fae48da919741cda24b0b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846eea04fb754f439ea1da73c0ffa2ec"
I20260806 11:41:40.064657 3378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f885bbce828414f86d67332b7cd6dbb" candidate_uuid: "99be8fe64fae48da919741cda24b0b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08"
I20260806 11:41:40.064778 3378 raft_consensus.cc:3037] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:40.064801 3569 raft_consensus.cc:3037] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:40.065841 3569 raft_consensus.cc:2454] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be8fe64fae48da919741cda24b0b84 in term 1.
I20260806 11:41:40.066004 3378 raft_consensus.cc:2454] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be8fe64fae48da919741cda24b0b84 in term 1.
I20260806 11:41:40.066129 3069 leader_election.cc:302] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 846eea04fb754f439ea1da73c0ffa2ec, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:40.066344 3677 raft_consensus.cc:2781] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:40.066596 3677 raft_consensus.cc:686] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 1 LEADER]: Becoming Leader. State: Replica: 99be8fe64fae48da919741cda24b0b84, State: Running, Role: LEADER
I20260806 11:41:40.066860 3677 consensus_queue.cc:237] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:40.069320 2944 catalog_manager.cc:5495] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99be8fe64fae48da919741cda24b0b84 (127.0.113.1). New cstate: current_term: 1 leader_uuid: "99be8fe64fae48da919741cda24b0b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } health_report { overall_health: UNKNOWN } } }
I20260806 11:41:40.093277 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:40.089538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.0.113.1:35727)
0806 11:41:40.089651 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:40.089655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:40.089681 (+ 26us) server_negotiation.cc:407] Connection header received
0806 11:41:40.089838 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:40.089840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:40.089897 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:40.089982 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:40.090256 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.090742 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.091882 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.092076 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.092147 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:40.092455 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:40.092857 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:40.092943 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:40.093065 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 11:41:40.093109 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:41:40.104054 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:40.100666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:47789 (local address 127.0.113.3:41533)
0806 11:41:40.100805 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:40.100809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:40.101002 (+ 193us) server_negotiation.cc:407] Connection header received
0806 11:41:40.101155 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:40.101158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:40.101208 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:40.101288 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:40.101545 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.102068 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.102951 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.103160 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.103280 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:40.103603 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:40.103672 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:40.103753 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:40.103852 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 11:41:40.103910 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 11:41:40.104080 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:40.100519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41533 (local address 127.0.113.1:47789)
0806 11:41:40.100917 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:40.100929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:40.101046 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:40.101299 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:40.101302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:40.101310 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:41:40.101441 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:40.101447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.102177 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.102182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:40.102825 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:40.102833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.103281 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.103304 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:40.103409 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:40.103451 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:41:40.103760 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:41:40.103770 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 11:41:40.103832 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 11:41:40.103910 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":127,"threads_started":1}
I20260806 11:41:40.110697 3569 raft_consensus.cc:1261] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Refusing update from remote peer 99be8fe64fae48da919741cda24b0b84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:40.110782 3378 raft_consensus.cc:1261] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Refusing update from remote peer 99be8fe64fae48da919741cda24b0b84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:40.111172 3753 consensus_queue.cc:1035] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:40.111404 3677 consensus_queue.cc:1035] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:40.114902 3718 mvcc.cc:205] Tried to move back new op lower bound from 7315523584449032192 to 7315523584276631552. Current Snapshot: MvccSnapshot[applied={T|T < 7315523584449032192}]
I20260806 11:41:40.131834 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:40.127487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:41099 (local address 127.0.113.1:35727)
0806 11:41:40.127595 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:40.127600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:40.127915 (+ 315us) server_negotiation.cc:407] Connection header received
0806 11:41:40.128045 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:40.128048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:40.128094 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:40.128165 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:40.129017 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.129525 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.130685 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.130908 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.131015 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:40.131326 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:40.131439 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:40.131515 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:40.131656 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 11:41:40.131705 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 11:41:40.131834 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:40.127429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:35727 (local address 127.0.113.3:41099)
0806 11:41:40.127819 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:40.127834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:40.127957 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:40.128557 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:40.128560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:40.128568 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:41:40.128680 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:40.128689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.129610 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.129613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:40.130557 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:40.130567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:40.131020 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:40.131023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:40.131123 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:40.131173 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:41:40.131555 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:41:40.131561 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 11:41:40.131641 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 11:41:40.131721 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
W20260806 11:41:40.132884 3718 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 11:41:41.689347 2941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:41.689502 2941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:41.691798 2941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: Sending DeleteTablet for 3 replicas of tablet 5f885bbce828414f86d67332b7cd6dbb
I20260806 11:41:41.692371 3136 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f885bbce828414f86d67332b7cd6dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:41 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:58038
I20260806 11:41:41.692476 3351 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f885bbce828414f86d67332b7cd6dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:41 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:49850
I20260806 11:41:41.692821 3543 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f885bbce828414f86d67332b7cd6dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:41 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:44640
I20260806 11:41:41.693173 3797 tablet_replica.cc:330] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: stopping tablet replica
I20260806 11:41:41.693195 3796 tablet_replica.cc:330] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: stopping tablet replica
I20260806 11:41:41.693351 3796 raft_consensus.cc:2229] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:41.693384 3797 raft_consensus.cc:2229] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:41.693475 3796 raft_consensus.cc:2258] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:41.693593 3795 tablet_replica.cc:330] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: stopping tablet replica
I20260806 11:41:41.693634 3797 raft_consensus.cc:2258] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:41.693770 3795 raft_consensus.cc:2229] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:41:41.694062 3795 raft_consensus.cc:2258] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:41.694321 3797 ts_tablet_manager.cc:1905] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:41.694757 3795 ts_tablet_manager.cc:1905] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:41.696748 3796 ts_tablet_manager.cc:1905] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:41.697489 3795 ts_tablet_manager.cc:1918] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 11:41:41.697584 3795 log.cc:1196] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/5f885bbce828414f86d67332b7cd6dbb
I20260806 11:41:41.697566 3797 ts_tablet_manager.cc:1918] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 11:41:41.697643 3797 log.cc:1196] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/5f885bbce828414f86d67332b7cd6dbb
I20260806 11:41:41.697882 3795 ts_tablet_manager.cc:1939] T 5f885bbce828414f86d67332b7cd6dbb P 99be8fe64fae48da919741cda24b0b84: Deleting consensus metadata
I20260806 11:41:41.697962 3797 ts_tablet_manager.cc:1939] T 5f885bbce828414f86d67332b7cd6dbb P 846eea04fb754f439ea1da73c0ffa2ec: Deleting consensus metadata
I20260806 11:41:41.698590 2923 catalog_manager.cc:4841] TS 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727): tablet 5f885bbce828414f86d67332b7cd6dbb (table test-table [id=9504fa3ed0c048788f98cdbb770309a4]) successfully deleted
I20260806 11:41:41.698697 2921 catalog_manager.cc:4841] TS 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533): tablet 5f885bbce828414f86d67332b7cd6dbb (table test-table [id=9504fa3ed0c048788f98cdbb770309a4]) successfully deleted
I20260806 11:41:41.699980 3796 ts_tablet_manager.cc:1918] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 11:41:41.700050 3796 log.cc:1196] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/5f885bbce828414f86d67332b7cd6dbb
I20260806 11:41:41.700241 3796 ts_tablet_manager.cc:1939] T 5f885bbce828414f86d67332b7cd6dbb P 416c3182b1434e20a2419f908049bb08: Deleting consensus metadata
I20260806 11:41:41.700989 2923 catalog_manager.cc:4841] TS 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359): tablet 5f885bbce828414f86d67332b7cd6dbb (table test-table [id=9504fa3ed0c048788f98cdbb770309a4]) successfully deleted
Aug 06 11:41:41 dist-test-slave-45dx krb5kdc[2879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016501, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 11:41:41 dist-test-slave-45dx krb5kdc[2879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016501, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:41.724574 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:41.716231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.0.113.62:37385)
0806 11:41:41.716504 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:41.716522 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:41.716540 (+ 18us) server_negotiation.cc:407] Connection header received
0806 11:41:41.716664 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:41.716666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:41.716707 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:41.716790 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:41.717436 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:41.717977 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:41.718739 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:41.718960 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:41.719032 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:41.721399 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:41.721403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:41.721405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:41.721427 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:41.722940 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:41.723348 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:41.723356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:41.723357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:41.723436 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:41.723698 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:41.723703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:41.723705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:41.723887 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:41.724024 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:41.724248 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 11:41:41.724400 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":141,"threads_started":1}
I20260806 11:41:41.727005 2941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:41:41.727388 2941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:41.731671 3136 tablet_service.cc:1467] Processing CreateTablet for tablet 3e01da1ef87e48349b909ac507a5ff95 (DEFAULT_TABLE table=test-table [id=1a268260062c45e0883102469337fe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:41.731734 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 3e01da1ef87e48349b909ac507a5ff95 (DEFAULT_TABLE table=test-table [id=1a268260062c45e0883102469337fe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:41.731864 3543 tablet_service.cc:1467] Processing CreateTablet for tablet 3e01da1ef87e48349b909ac507a5ff95 (DEFAULT_TABLE table=test-table [id=1a268260062c45e0883102469337fe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:41.731990 3136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e01da1ef87e48349b909ac507a5ff95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:41.732038 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e01da1ef87e48349b909ac507a5ff95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:41.732139 3543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e01da1ef87e48349b909ac507a5ff95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:41.734059 3809 tablet_bootstrap.cc:492] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: Bootstrap starting.
I20260806 11:41:41.734617 3811 tablet_bootstrap.cc:492] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: Bootstrap starting.
I20260806 11:41:41.735069 3812 tablet_bootstrap.cc:492] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: Bootstrap starting.
I20260806 11:41:41.735219 3809 tablet_bootstrap.cc:654] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:41.735493 3811 tablet_bootstrap.cc:654] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:41.736193 3812 tablet_bootstrap.cc:654] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:41.736631 3809 tablet_bootstrap.cc:492] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: No bootstrap required, opened a new log
I20260806 11:41:41.736739 3809 ts_tablet_manager.cc:1397] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 11:41:41.736884 3811 tablet_bootstrap.cc:492] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: No bootstrap required, opened a new log
I20260806 11:41:41.736996 3811 ts_tablet_manager.cc:1397] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 11:41:41.737229 3809 raft_consensus.cc:348] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:41.737383 3809 raft_consensus.cc:374] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:41.737430 3809 raft_consensus.cc:729] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416c3182b1434e20a2419f908049bb08, State: Initialized, Role: FOLLOWER
I20260806 11:41:41.737560 3809 consensus_queue.cc:260] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:41.737617 3812 tablet_bootstrap.cc:492] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: No bootstrap required, opened a new log
I20260806 11:41:41.737654 3811 raft_consensus.cc:348] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:41.737778 3811 raft_consensus.cc:374] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:41.737820 3811 raft_consensus.cc:729] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846eea04fb754f439ea1da73c0ffa2ec, State: Initialized, Role: FOLLOWER
I20260806 11:41:41.737974 3812 ts_tablet_manager.cc:1397] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 11:41:41.737977 3809 ts_tablet_manager.cc:1428] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:41.737960 3811 consensus_queue.cc:260] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:41.738260 3811 ts_tablet_manager.cc:1428] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 11:41:41.738642 3812 raft_consensus.cc:348] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:41.738792 3812 raft_consensus.cc:374] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:41.738842 3812 raft_consensus.cc:729] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be8fe64fae48da919741cda24b0b84, State: Initialized, Role: FOLLOWER
I20260806 11:41:41.738986 3812 consensus_queue.cc:260] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:41.739318 3812 ts_tablet_manager.cc:1428] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:42.055316 3769 raft_consensus.cc:482] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:42.055483 3769 raft_consensus.cc:504] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.056005 3769 leader_election.cc:288] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727), 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359)
I20260806 11:41:42.056485 3166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01da1ef87e48349b909ac507a5ff95" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be8fe64fae48da919741cda24b0b84" is_pre_election: true
I20260806 11:41:42.056572 3378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01da1ef87e48349b909ac507a5ff95" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08" is_pre_election: true
I20260806 11:41:42.056654 3166 raft_consensus.cc:2454] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 0.
I20260806 11:41:42.056697 3378 raft_consensus.cc:2454] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 0.
I20260806 11:41:42.057042 3475 leader_election.cc:302] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 846eea04fb754f439ea1da73c0ffa2ec, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:42.057265 3769 raft_consensus.cc:2781] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:42.057330 3769 raft_consensus.cc:482] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:42.057368 3769 raft_consensus.cc:3037] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:42.058429 3769 raft_consensus.cc:504] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.058796 3769 leader_election.cc:288] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 election: Requested vote from peers 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727), 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359)
I20260806 11:41:42.059194 3166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01da1ef87e48349b909ac507a5ff95" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be8fe64fae48da919741cda24b0b84"
I20260806 11:41:42.059342 3166 raft_consensus.cc:3037] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:42.059320 3378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01da1ef87e48349b909ac507a5ff95" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08"
I20260806 11:41:42.059453 3378 raft_consensus.cc:3037] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:42.060616 3378 raft_consensus.cc:2454] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 1.
I20260806 11:41:42.060616 3166 raft_consensus.cc:2454] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 1.
I20260806 11:41:42.060943 3475 leader_election.cc:302] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 846eea04fb754f439ea1da73c0ffa2ec, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:42.061172 3769 raft_consensus.cc:2781] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:42.061281 3769 raft_consensus.cc:686] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 LEADER]: Becoming Leader. State: Replica: 846eea04fb754f439ea1da73c0ffa2ec, State: Running, Role: LEADER
I20260806 11:41:42.061427 3769 consensus_queue.cc:237] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.063347 2943 catalog_manager.cc:5495] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3). New cstate: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } health_report { overall_health: HEALTHY } } }
I20260806 11:41:42.108124 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:42.104586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.0.113.3:41533)
0806 11:41:42.105029 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:42.105036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:42.105046 (+ 10us) server_negotiation.cc:407] Connection header received
0806 11:41:42.105082 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:42.105084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:42.105124 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:42.105219 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:42.105499 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:42.106047 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:42.106980 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:42.107225 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:42.107381 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:42.107676 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:42.107749 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:42.107826 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:42.107916 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 11:41:42.107970 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":107,"threads_started":1}
I20260806 11:41:42.118175 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:42.114082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41533 (local address 127.0.113.3:36899)
0806 11:41:42.114433 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:42.114446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:42.114531 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:42.114783 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:42.114785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:42.114791 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:41:42.114898 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:42.114902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:42.116058 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:42.116060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:42.116712 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:42.116722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:42.117398 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:42.117401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:42.117505 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:42.117530 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:41:42.117883 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:41:42.117890 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 11:41:42.117956 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 11:41:42.118023 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":31,"thread_start_us":138,"threads_started":1}
I20260806 11:41:42.118315 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:42.114257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:36899 (local address 127.0.113.3:41533)
0806 11:41:42.114408 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:42.114415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:42.114516 (+ 101us) server_negotiation.cc:407] Connection header received
0806 11:41:42.114603 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:42.114609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:42.114679 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:42.114777 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:42.115001 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:42.115925 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:42.116833 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:42.117262 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:42.117428 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:42.117649 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:42.117745 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:42.117828 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:42.117991 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 11:41:42.118066 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":46}
I20260806 11:41:42.125144 3166 raft_consensus.cc:1261] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Refusing update from remote peer 846eea04fb754f439ea1da73c0ffa2ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:42.125216 3378 raft_consensus.cc:1261] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Refusing update from remote peer 846eea04fb754f439ea1da73c0ffa2ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:42.125592 3708 consensus_queue.cc:1035] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:42.125733 3769 consensus_queue.cc:1035] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:42.164419 2943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:42.164562 2943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:42.166561 2943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: Sending DeleteTablet for 3 replicas of tablet 3e01da1ef87e48349b909ac507a5ff95
I20260806 11:41:42.167125 3136 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e01da1ef87e48349b909ac507a5ff95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:42 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:58038
I20260806 11:41:42.167136 3542 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e01da1ef87e48349b909ac507a5ff95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:42 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:44640
I20260806 11:41:42.167320 3797 tablet_replica.cc:330] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: stopping tablet replica
I20260806 11:41:42.167457 3797 raft_consensus.cc:2229] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:41:42.167460 3351 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e01da1ef87e48349b909ac507a5ff95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:42 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:49850
I20260806 11:41:42.167711 3797 raft_consensus.cc:2258] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:42.167712 3796 tablet_replica.cc:330] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: stopping tablet replica
I20260806 11:41:42.167896 3795 tablet_replica.cc:330] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: stopping tablet replica
I20260806 11:41:42.167888 3796 raft_consensus.cc:2229] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:42.168005 3795 raft_consensus.cc:2229] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:42.168008 3796 raft_consensus.cc:2258] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:42.168084 3795 raft_consensus.cc:2258] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:42.168748 3796 ts_tablet_manager.cc:1905] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:42.168749 3797 ts_tablet_manager.cc:1905] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:42.171306 3796 ts_tablet_manager.cc:1918] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:41:42.171306 3797 ts_tablet_manager.cc:1918] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:41:42.171394 3797 log.cc:1196] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/3e01da1ef87e48349b909ac507a5ff95
I20260806 11:41:42.171394 3796 log.cc:1196] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/3e01da1ef87e48349b909ac507a5ff95
I20260806 11:41:42.171680 3796 ts_tablet_manager.cc:1939] T 3e01da1ef87e48349b909ac507a5ff95 P 416c3182b1434e20a2419f908049bb08: Deleting consensus metadata
I20260806 11:41:42.171731 3797 ts_tablet_manager.cc:1939] T 3e01da1ef87e48349b909ac507a5ff95 P 846eea04fb754f439ea1da73c0ffa2ec: Deleting consensus metadata
I20260806 11:41:42.172487 2921 catalog_manager.cc:4841] TS 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533): tablet 3e01da1ef87e48349b909ac507a5ff95 (table test-table [id=1a268260062c45e0883102469337fe60]) successfully deleted
I20260806 11:41:42.172561 2923 catalog_manager.cc:4841] TS 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359): tablet 3e01da1ef87e48349b909ac507a5ff95 (table test-table [id=1a268260062c45e0883102469337fe60]) successfully deleted
I20260806 11:41:42.173596 3795 ts_tablet_manager.cc:1905] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:42.175851 3795 ts_tablet_manager.cc:1918] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:41:42.175951 3795 log.cc:1196] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/3e01da1ef87e48349b909ac507a5ff95
I20260806 11:41:42.176227 3795 ts_tablet_manager.cc:1939] T 3e01da1ef87e48349b909ac507a5ff95 P 99be8fe64fae48da919741cda24b0b84: Deleting consensus metadata
I20260806 11:41:42.176878 2923 catalog_manager.cc:4841] TS 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727): tablet 3e01da1ef87e48349b909ac507a5ff95 (table test-table [id=1a268260062c45e0883102469337fe60]) successfully deleted
I20260806 11:41:42.177168 2943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:41:42.177592 2943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:42.182200 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a (DEFAULT_TABLE table=test-table [id=e6e547e932c54310b8431e2dd27181f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:42.182258 3542 tablet_service.cc:1467] Processing CreateTablet for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a (DEFAULT_TABLE table=test-table [id=e6e547e932c54310b8431e2dd27181f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:42.182504 3542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:42.182504 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:42.184852 3811 tablet_bootstrap.cc:492] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: Bootstrap starting.
I20260806 11:41:42.185639 3811 tablet_bootstrap.cc:654] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:42.186198 3809 tablet_bootstrap.cc:492] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: Bootstrap starting.
I20260806 11:41:42.186875 3809 tablet_bootstrap.cc:654] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:42.187531 3136 tablet_service.cc:1467] Processing CreateTablet for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a (DEFAULT_TABLE table=test-table [id=e6e547e932c54310b8431e2dd27181f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:42.187819 3136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:42.189864 3812 tablet_bootstrap.cc:492] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84: Bootstrap starting.
I20260806 11:41:42.190267 3809 tablet_bootstrap.cc:492] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: No bootstrap required, opened a new log
I20260806 11:41:42.190373 3809 ts_tablet_manager.cc:1397] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 11:41:42.190692 3812 tablet_bootstrap.cc:654] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:42.190791 3809 raft_consensus.cc:348] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.190927 3809 raft_consensus.cc:374] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:42.190976 3809 raft_consensus.cc:729] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416c3182b1434e20a2419f908049bb08, State: Initialized, Role: FOLLOWER
I20260806 11:41:42.191110 3809 consensus_queue.cc:260] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.191433 3809 ts_tablet_manager.cc:1428] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:42.191946 3811 tablet_bootstrap.cc:492] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: No bootstrap required, opened a new log
I20260806 11:41:42.192036 3811 ts_tablet_manager.cc:1397] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260806 11:41:42.192451 3811 raft_consensus.cc:348] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.192589 3811 raft_consensus.cc:374] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:42.192628 3811 raft_consensus.cc:729] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846eea04fb754f439ea1da73c0ffa2ec, State: Initialized, Role: FOLLOWER
I20260806 11:41:42.192762 3811 consensus_queue.cc:260] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.193037 3811 ts_tablet_manager.cc:1428] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:42.194583 3812 tablet_bootstrap.cc:492] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84: No bootstrap required, opened a new log
I20260806 11:41:42.194674 3812 ts_tablet_manager.cc:1397] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260806 11:41:42.195101 3812 raft_consensus.cc:348] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.195233 3812 raft_consensus.cc:374] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:42.195274 3812 raft_consensus.cc:729] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be8fe64fae48da919741cda24b0b84, State: Initialized, Role: FOLLOWER
I20260806 11:41:42.195390 3812 consensus_queue.cc:260] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.195662 3812 ts_tablet_manager.cc:1428] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:41:42.419400 3708 raft_consensus.cc:482] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:42.419559 3708 raft_consensus.cc:504] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.420004 3708 leader_election.cc:288] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727), 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359)
I20260806 11:41:42.420511 3164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cff6fe9a0f6431ba8507b2125ce2f8a" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be8fe64fae48da919741cda24b0b84" is_pre_election: true
I20260806 11:41:42.420523 3378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cff6fe9a0f6431ba8507b2125ce2f8a" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08" is_pre_election: true
I20260806 11:41:42.420687 3164 raft_consensus.cc:2454] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 0.
I20260806 11:41:42.420712 3378 raft_consensus.cc:2454] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 0.
I20260806 11:41:42.421023 3475 leader_election.cc:302] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 846eea04fb754f439ea1da73c0ffa2ec, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:42.421208 3708 raft_consensus.cc:2781] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:42.421268 3708 raft_consensus.cc:482] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:42.421319 3708 raft_consensus.cc:3037] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:42.422498 3708 raft_consensus.cc:504] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.422770 3708 leader_election.cc:288] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 election: Requested vote from peers 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727), 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359)
I20260806 11:41:42.423133 3164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cff6fe9a0f6431ba8507b2125ce2f8a" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be8fe64fae48da919741cda24b0b84"
I20260806 11:41:42.423192 3378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cff6fe9a0f6431ba8507b2125ce2f8a" candidate_uuid: "846eea04fb754f439ea1da73c0ffa2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416c3182b1434e20a2419f908049bb08"
I20260806 11:41:42.423264 3164 raft_consensus.cc:3037] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:42.423295 3378 raft_consensus.cc:3037] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:42.424171 3378 raft_consensus.cc:2454] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 1.
I20260806 11:41:42.424185 3164 raft_consensus.cc:2454] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846eea04fb754f439ea1da73c0ffa2ec in term 1.
I20260806 11:41:42.424469 3475 leader_election.cc:302] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 846eea04fb754f439ea1da73c0ffa2ec, 99be8fe64fae48da919741cda24b0b84; no voters:
I20260806 11:41:42.424661 3708 raft_consensus.cc:2781] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:42.424777 3708 raft_consensus.cc:686] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 1 LEADER]: Becoming Leader. State: Replica: 846eea04fb754f439ea1da73c0ffa2ec, State: Running, Role: LEADER
I20260806 11:41:42.424907 3708 consensus_queue.cc:237] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } }
I20260806 11:41:42.426553 2939 catalog_manager.cc:5495] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3). New cstate: current_term: 1 leader_uuid: "846eea04fb754f439ea1da73c0ffa2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846eea04fb754f439ea1da73c0ffa2ec" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41533 } health_report { overall_health: HEALTHY } } }
I20260806 11:41:42.492008 3164 raft_consensus.cc:1261] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84 [term 1 FOLLOWER]: Refusing update from remote peer 846eea04fb754f439ea1da73c0ffa2ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:42.491986 3378 raft_consensus.cc:1261] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Refusing update from remote peer 846eea04fb754f439ea1da73c0ffa2ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:42.492411 3781 consensus_queue.cc:1035] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be8fe64fae48da919741cda24b0b84" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:42.492573 3708 consensus_queue.cc:1035] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "416c3182b1434e20a2419f908049bb08" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 11:41:42.505818 3767 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 11:41:44.064550 2943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:44.064702 2943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:44.066898 2943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75743732347a4b06bdd4c017310bc74b: Sending DeleteTablet for 3 replicas of tablet 0cff6fe9a0f6431ba8507b2125ce2f8a
I20260806 11:41:44.067404 3136 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:44 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:58038
I20260806 11:41:44.067472 3351 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:44 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:49850
I20260806 11:41:44.067523 3543 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:44 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:44640
I20260806 11:41:44.067831 3866 tablet_replica.cc:330] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 99be8fe64fae48da919741cda24b0b84: stopping tablet replica
I20260806 11:41:44.067890 3867 tablet_replica.cc:330] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: stopping tablet replica
I20260806 11:41:44.067894 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 3049
I20260806 11:41:44.071486 3867 raft_consensus.cc:2229] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:44.071641 3867 raft_consensus.cc:2258] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:44.071863 3868 tablet_replica.cc:330] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: stopping tablet replica
I20260806 11:41:44.071973 3868 raft_consensus.cc:2229] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:41:44.072144 3868 raft_consensus.cc:2258] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:44.073799 3868 ts_tablet_manager.cc:1905] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:44.076566 3868 ts_tablet_manager.cc:1918] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 11:41:44.076658 3868 log.cc:1196] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/0cff6fe9a0f6431ba8507b2125ce2f8a
I20260806 11:41:44.076953 3868 ts_tablet_manager.cc:1939] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 846eea04fb754f439ea1da73c0ffa2ec: Deleting consensus metadata
I20260806 11:41:44.077185 3867 ts_tablet_manager.cc:1905] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:44.077627 2921 catalog_manager.cc:4841] TS 846eea04fb754f439ea1da73c0ffa2ec (127.0.113.3:41533): tablet 0cff6fe9a0f6431ba8507b2125ce2f8a (table test-table [id=e6e547e932c54310b8431e2dd27181f8]) successfully deleted
I20260806 11:41:44.080547 3867 ts_tablet_manager.cc:1918] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 11:41:44.080631 3867 log.cc:1196] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/0cff6fe9a0f6431ba8507b2125ce2f8a
I20260806 11:41:44.080907 3867 ts_tablet_manager.cc:1939] T 0cff6fe9a0f6431ba8507b2125ce2f8a P 416c3182b1434e20a2419f908049bb08: Deleting consensus metadata
I20260806 11:41:44.081607 2923 catalog_manager.cc:4841] TS 416c3182b1434e20a2419f908049bb08 (127.0.113.2:38359): tablet 0cff6fe9a0f6431ba8507b2125ce2f8a (table test-table [id=e6e547e932c54310b8431e2dd27181f8]) successfully deleted
W20260806 11:41:44.084956 2923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:35727 (error 108)
I20260806 11:41:44.085202 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 3259
I20260806 11:41:44.091763 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:44.091173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:35727 (local address 127.0.0.1:42712)
0806 11:41:44.091542 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:44.091631 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":93,"threads_started":1}
W20260806 11:41:44.091900 2923 catalog_manager.cc:4584] TS 99be8fe64fae48da919741cda24b0b84 (127.0.113.1:35727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0cff6fe9a0f6431ba8507b2125ce2f8a: Network error: Client connection negotiation failed: client connection to 127.0.113.1:35727: connect: Connection refused (error 111)
I20260806 11:41:44.098033 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 3453
I20260806 11:41:44.108588 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 2900
2026-08-06T11:41:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7094 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:44 dist-test-slave-45dx krb5kdc[3874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:44 dist-test-slave-45dx krb5kdc[3874](info): set up 2 sockets
Aug 06 11:41:44 dist-test-slave-45dx krb5kdc[3874](info): commencing operation
krb5kdc: starting...
W20260806 11:41:45.319607 452 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:45 dist-test-slave-45dx krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:45.441241 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:43857
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:43857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:45.551856 3894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:45.552147 3894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:45.552222 3894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:45.556133 3894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:45.556200 3894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:45.556237 3894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:45.556304 3894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:45.556367 3894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:45.561499 3894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:43857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:43857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:45.562523 3894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:45.563509 3894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:45.572031 3901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:45.572194 3894 server_base.cc:1034] running on GCE node
W20260806 11:41:45.572142 3900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:45.572142 3906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:45.572958 3894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:45.574002 3894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:45.575176 3894 hybrid_clock.cc:648] HybridClock initialized: now 1786016505575146 us; error 43 us; skew 500 ppm
Aug 06 11:41:45 dist-test-slave-45dx krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:45.578101 3894 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:45.578922 3894 webserver.cc:460] Webserver started at http://127.0.113.62:33833/ using document root <none> and password file <none>
I20260806 11:41:45.579420 3894 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:45.579479 3894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:45.579638 3894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:45.581790 3894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "e9d9310c516e4365ade98feaae70db49"
format_stamp: "Formatted at 2026-08-06 11:41:45 on dist-test-slave-45dx"
server_key: "31d323ce91d8bd67d0145988041cd25d"
server_key_iv: "8d9e2e4499e97d26a2ca74121e4dfc1c"
server_key_version: "encryptionkey@0"
I20260806 11:41:45.582321 3894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "e9d9310c516e4365ade98feaae70db49"
format_stamp: "Formatted at 2026-08-06 11:41:45 on dist-test-slave-45dx"
server_key: "31d323ce91d8bd67d0145988041cd25d"
server_key_iv: "8d9e2e4499e97d26a2ca74121e4dfc1c"
server_key_version: "encryptionkey@0"
I20260806 11:41:45.585837 3894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 11:41:45.588063 3914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:45.589241 3894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:45.589372 3894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "e9d9310c516e4365ade98feaae70db49"
format_stamp: "Formatted at 2026-08-06 11:41:45 on dist-test-slave-45dx"
server_key: "31d323ce91d8bd67d0145988041cd25d"
server_key_iv: "8d9e2e4499e97d26a2ca74121e4dfc1c"
server_key_version: "encryptionkey@0"
I20260806 11:41:45.589470 3894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:45.603698 3894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:45.606392 3894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:45.606585 3894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:45.613360 3894 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:43857
I20260806 11:41:45.613372 3983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:43857 every 8 connection(s)
I20260806 11:41:45.614239 3894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:45.616989 3984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:45.617054 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 3894
I20260806 11:41:45.617157 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:45.617481 452 external_mini_cluster.cc:1428] Setting key 1bf909e4bbf2974dfa3e73a22e36f877
I20260806 11:41:45.623647 3984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49: Bootstrap starting.
I20260806 11:41:45.626228 3984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49: Neither blocks nor log segments found. Creating new log.
Aug 06 11:41:45 dist-test-slave-45dx krb5kdc[3874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:45.627143 3984 log.cc:824] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:45.629211 3984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49: No bootstrap required, opened a new log
I20260806 11:41:45.631505 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:45.618807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.0.113.62:43857)
0806 11:41:45.619205 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:45.619220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:45.619255 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:41:45.620136 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:45.620154 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:45.620523 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:45.620795 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:45.621676 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:45.622628 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:45.623757 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:45.624200 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:45.624380 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:45.627026 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:45.627031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:45.627042 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:45.627068 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:45.628848 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:45.629341 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:45.629347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:45.629353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:45.629473 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:45.629765 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:45.629768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:45.629769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:45.630104 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:45.630238 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:45.630521 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 11:41:45.630823 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
I20260806 11:41:45.633042 3984 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d9310c516e4365ade98feaae70db49" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43857 } }
I20260806 11:41:45.633291 3984 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:45.633390 3984 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d9310c516e4365ade98feaae70db49, State: Initialized, Role: FOLLOWER
I20260806 11:41:45.633800 3984 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d9310c516e4365ade98feaae70db49" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43857 } }
I20260806 11:41:45.633924 3984 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:45.633970 3984 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:45.634040 3984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:45.634877 3984 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d9310c516e4365ade98feaae70db49" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43857 } }
I20260806 11:41:45.635181 3984 leader_election.cc:302] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d9310c516e4365ade98feaae70db49; no voters:
I20260806 11:41:45.635432 3984 leader_election.cc:288] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:45.635550 3991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:45.635776 3991 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [term 1 LEADER]: Becoming Leader. State: Replica: e9d9310c516e4365ade98feaae70db49, State: Running, Role: LEADER
I20260806 11:41:45.636092 3991 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d9310c516e4365ade98feaae70db49" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43857 } }
I20260806 11:41:45.636654 3984 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:45.637849 3993 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d9310c516e4365ade98feaae70db49. Latest consensus state: current_term: 1 leader_uuid: "e9d9310c516e4365ade98feaae70db49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d9310c516e4365ade98feaae70db49" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43857 } } }
I20260806 11:41:45.638032 3993 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:45.637878 3992 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d9310c516e4365ade98feaae70db49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d9310c516e4365ade98feaae70db49" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43857 } } }
I20260806 11:41:45.638257 3992 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:45.638969 4006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:45.641609 4006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:45.646553 4006 catalog_manager.cc:1347] Generated new cluster ID: 740f1c8a574a4fca82dcb66c5a5716f1
I20260806 11:41:45.646610 4006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:45.655614 4006 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:45.656735 4006 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:45.668900 4006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9d9310c516e4365ade98feaae70db49: Generated new TSK 0
I20260806 11:41:45.669631 4006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:41:45.701478 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:43857
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:45.810158 4021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:45.810420 4021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:45.810493 4021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:45.814301 4021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:45.814368 4021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:45.814453 4021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:45.819324 4021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:43857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:45.820266 4021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:45.821233 4021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:45.830850 4027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:45.830894 4029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:45.830847 4032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:45.831176 4021 server_base.cc:1034] running on GCE node
I20260806 11:41:45.831496 4021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:45.832051 4021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:45.833200 4021 hybrid_clock.cc:648] HybridClock initialized: now 1786016505833160 us; error 50 us; skew 500 ppm
Aug 06 11:41:45 dist-test-slave-45dx krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:45.836058 4021 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:41:45.836982 4021 webserver.cc:460] Webserver started at http://127.0.113.1:39003/ using document root <none> and password file <none>
I20260806 11:41:45.837513 4021 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:45.837574 4021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:45.837731 4021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:45.839761 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "e6ce5dba37bd4a3f8e5fae2d07a29e1e"
format_stamp: "Formatted at 2026-08-06 11:41:45 on dist-test-slave-45dx"
server_key: "1c4424da10d3498891f04d85e0e43ad3"
server_key_iv: "ce4a1c110464c2a9c02bc8feaafbe534"
server_key_version: "encryptionkey@0"
I20260806 11:41:45.840284 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "e6ce5dba37bd4a3f8e5fae2d07a29e1e"
format_stamp: "Formatted at 2026-08-06 11:41:45 on dist-test-slave-45dx"
server_key: "1c4424da10d3498891f04d85e0e43ad3"
server_key_iv: "ce4a1c110464c2a9c02bc8feaafbe534"
server_key_version: "encryptionkey@0"
I20260806 11:41:45.843818 4021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:41:45.846105 4043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:45.847272 4021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:45.847404 4021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "e6ce5dba37bd4a3f8e5fae2d07a29e1e"
format_stamp: "Formatted at 2026-08-06 11:41:45 on dist-test-slave-45dx"
server_key: "1c4424da10d3498891f04d85e0e43ad3"
server_key_iv: "ce4a1c110464c2a9c02bc8feaafbe534"
server_key_version: "encryptionkey@0"
I20260806 11:41:45.847508 4021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:45.857055 4021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:45.859803 4021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:45.859990 4021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:45.860551 4021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:45.861325 4021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:45.861397 4021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:45.861462 4021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:45.861506 4021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:45.870090 4021 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:35851
I20260806 11:41:45.870121 4205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:35851 every 8 connection(s)
I20260806 11:41:45.870864 4021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:45.877395 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 4021
I20260806 11:41:45.877538 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:45.877840 452 external_mini_cluster.cc:1428] Setting key 366e0ef03af963a2bbda67afcace10f9
Aug 06 11:41:45 dist-test-slave-45dx krb5kdc[3874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:45.884112 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:45.872434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:33601 (local address 127.0.113.62:43857)
0806 11:41:45.872556 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:45.872559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:45.873277 (+ 718us) server_negotiation.cc:407] Connection header received
0806 11:41:45.874704 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:45.874707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:45.874753 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:45.874826 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:45.875864 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:45.876440 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:45.877499 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:45.877867 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:45.877976 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:45.880282 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:45.880288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:45.880294 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:45.880349 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:45.881812 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:45.882448 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:45.882451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:45.882452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:45.882527 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:45.882876 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:45.882880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:45.882882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:45.883154 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:45.883253 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:45.883690 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 11:41:45.883857 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 11:41:45.884737 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:45.872724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43857 (local address 127.0.113.1:33601)
0806 11:41:45.873132 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:45.873170 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:45.874522 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:45.874993 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:45.874999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:45.875325 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:45.875742 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:45.875754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:45.876537 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:45.876539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:45.877376 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:45.877384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:45.878035 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:45.878038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:45.878176 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:45.878731 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:45.878747 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:45.880130 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:45.881951 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:45.881960 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:45.881975 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:45.882347 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:45.882610 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:45.882618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:45.882620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:45.882778 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:45.883251 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:45.883255 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:45.883471 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 11:41:45.883611 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 11:41:45.883825 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":106,"threads_started":1}
I20260806 11:41:45.885840 4207 heartbeater.cc:347] Connected to a master server at 127.0.113.62:43857
I20260806 11:41:45.886025 4207 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:45.886442 4207 heartbeater.cc:511] Master 127.0.113.62:43857 requested a full tablet report, sending...
I20260806 11:41:45.887811 3939 ts_manager.cc:194] Registered new tserver with Master: e6ce5dba37bd4a3f8e5fae2d07a29e1e (127.0.113.1:35851)
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:41:45.888888 3939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:33601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:41:45.906867 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:43857
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:46.036029 4216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:46.036588 4216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:46.036667 4216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:46.044950 4216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:46.045030 4216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:46.045130 4216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:41:46.055016 4216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:43857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:46.056984 4216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:46.058723 4216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:46.077356 4224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:46.077589 4222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:46.077936 4216 server_base.cc:1034] running on GCE node
W20260806 11:41:46.078051 4227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:46.078971 4216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:46.080161 4216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:46.083526 4216 hybrid_clock.cc:648] HybridClock initialized: now 1786016506083404 us; error 129 us; skew 500 ppm
Aug 06 11:41:46 dist-test-slave-45dx krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:46.089529 4216 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:41:46.091472 4216 webserver.cc:460] Webserver started at http://127.0.113.2:35707/ using document root <none> and password file <none>
I20260806 11:41:46.092625 4216 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:46.092803 4216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:46.093284 4216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:46.097950 4216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "1e9a2885013d46fbbee737ab3b341544"
format_stamp: "Formatted at 2026-08-06 11:41:46 on dist-test-slave-45dx"
server_key: "4285d5c99a6630e6ed3415a2b6034deb"
server_key_iv: "0f0b21b79bb8613fcc224453c5e6490d"
server_key_version: "encryptionkey@0"
I20260806 11:41:46.099874 4216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "1e9a2885013d46fbbee737ab3b341544"
format_stamp: "Formatted at 2026-08-06 11:41:46 on dist-test-slave-45dx"
server_key: "4285d5c99a6630e6ed3415a2b6034deb"
server_key_iv: "0f0b21b79bb8613fcc224453c5e6490d"
server_key_version: "encryptionkey@0"
I20260806 11:41:46.110332 4216 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260806 11:41:46.115695 4237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:46.118417 4216 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 11:41:46.118637 4216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "1e9a2885013d46fbbee737ab3b341544"
format_stamp: "Formatted at 2026-08-06 11:41:46 on dist-test-slave-45dx"
server_key: "4285d5c99a6630e6ed3415a2b6034deb"
server_key_iv: "0f0b21b79bb8613fcc224453c5e6490d"
server_key_version: "encryptionkey@0"
I20260806 11:41:46.118817 4216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:46.143522 4216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:46.151175 4216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:46.151728 4216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:46.153103 4216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:46.155004 4216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:46.155117 4216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:46.155272 4216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:46.155457 4216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:46.175833 4216 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:34399
I20260806 11:41:46.176007 4395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:34399 every 8 connection(s)
I20260806 11:41:46.177747 4216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:41:46.186208 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 4216
I20260806 11:41:46.186427 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:41:46.186960 452 external_mini_cluster.cc:1428] Setting key 68afffe3b04c1accc71e3f889c2967c1
Aug 06 11:41:46 dist-test-slave-45dx krb5kdc[3874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:46.211241 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:46.181190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:49119 (local address 127.0.113.62:43857)
0806 11:41:46.181377 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:46.181383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:46.183050 (+ 1667us) server_negotiation.cc:407] Connection header received
0806 11:41:46.185772 (+ 2722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:46.185780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:46.185996 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:46.186198 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:46.188855 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:46.189908 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:46.194024 (+ 4116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:46.194868 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:46.195186 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:46.201559 (+ 6373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:46.201564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:46.201566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:46.201608 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:46.206106 (+ 4498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:46.207202 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:46.207207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:46.207209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:46.207479 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:46.208205 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:46.208221 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:46.208236 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:46.209022 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:46.209395 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:46.210333 (+ 938us) server_negotiation.cc:299] Negotiation successful
0806 11:41:46.210734 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:41:46.212551 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:46.182065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43857 (local address 127.0.113.2:49119)
0806 11:41:46.182728 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:46.182815 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:46.185317 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:46.186517 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:46.186536 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:46.187406 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:46.188569 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:46.188604 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:46.190125 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:46.190129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:46.193631 (+ 3502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:46.193673 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:46.195216 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:46.195220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:46.195551 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:46.196777 (+ 1226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:46.196897 (+ 120us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:46.201101 (+ 4204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:46.206359 (+ 5258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:46.206367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:46.206386 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:46.206998 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:46.207674 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:46.207678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:46.207679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:46.208051 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:46.209292 (+ 1241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:46.209329 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:46.209842 (+ 513us) client_negotiation.cc:769] Sending connection context
0806 11:41:46.210098 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 11:41:46.210654 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":183,"threads_started":1}
I20260806 11:41:46.215070 4396 heartbeater.cc:347] Connected to a master server at 127.0.113.62:43857
I20260806 11:41:46.215595 4396 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
I20260806 11:41:46.216881 4396 heartbeater.cc:511] Master 127.0.113.62:43857 requested a full tablet report, sending...
I20260806 11:41:46.219704 3939 ts_manager.cc:194] Registered new tserver with Master: 1e9a2885013d46fbbee737ab3b341544 (127.0.113.2:34399)
I20260806 11:41:46.221705 3939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:49119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:41:46.239184 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:43857
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:46.351811 4406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:46.352087 4406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:46.352180 4406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:46.356173 4406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:46.356254 4406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:46.356406 4406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:41:46.361392 4406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:43857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:46.362391 4406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:46.363371 4406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:46.372623 4416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:46.372637 4415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:46.372867 4406 server_base.cc:1034] running on GCE node
W20260806 11:41:46.373027 4418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:46.373447 4406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:46.374032 4406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:46.375204 4406 hybrid_clock.cc:648] HybridClock initialized: now 1786016506375173 us; error 47 us; skew 500 ppm
Aug 06 11:41:46 dist-test-slave-45dx krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:46.377978 4406 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:41:46.378818 4406 webserver.cc:460] Webserver started at http://127.0.113.3:42239/ using document root <none> and password file <none>
I20260806 11:41:46.379320 4406 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:46.379379 4406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:46.379539 4406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:46.381515 4406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "0d8767e2dc04409098be7c0186225df8"
format_stamp: "Formatted at 2026-08-06 11:41:46 on dist-test-slave-45dx"
server_key: "cee33fb79d0f20ee26752dd44fb16e2e"
server_key_iv: "5af9ea6b336ae7dbdb1a9404f47bb192"
server_key_version: "encryptionkey@0"
I20260806 11:41:46.382061 4406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "0d8767e2dc04409098be7c0186225df8"
format_stamp: "Formatted at 2026-08-06 11:41:46 on dist-test-slave-45dx"
server_key: "cee33fb79d0f20ee26752dd44fb16e2e"
server_key_iv: "5af9ea6b336ae7dbdb1a9404f47bb192"
server_key_version: "encryptionkey@0"
I20260806 11:41:46.385561 4406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:41:46.388120 4429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:46.389406 4406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:46.389535 4406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "0d8767e2dc04409098be7c0186225df8"
format_stamp: "Formatted at 2026-08-06 11:41:46 on dist-test-slave-45dx"
server_key: "cee33fb79d0f20ee26752dd44fb16e2e"
server_key_iv: "5af9ea6b336ae7dbdb1a9404f47bb192"
server_key_version: "encryptionkey@0"
I20260806 11:41:46.389638 4406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:46.404839 4406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:46.408026 4406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:46.408232 4406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:46.408886 4406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:46.409699 4406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:46.409746 4406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:46.409814 4406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:46.409854 4406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:46.418655 4406 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:36513
I20260806 11:41:46.418684 4585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:36513 every 8 connection(s)
I20260806 11:41:46.419489 4406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:41:46.425204 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 4406
I20260806 11:41:46.425326 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:41:46.425601 452 external_mini_cluster.cc:1428] Setting key e4c9159db7250ac40c5f07fe659b4404
Aug 06 11:41:46 dist-test-slave-45dx krb5kdc[3874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:46.433212 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:46.421326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:36903 (local address 127.0.113.62:43857)
0806 11:41:46.421460 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:46.421463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:46.422242 (+ 779us) server_negotiation.cc:407] Connection header received
0806 11:41:46.423698 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:46.423702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:46.423759 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:46.423862 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:46.425024 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:46.425616 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:46.427017 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:46.427325 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:46.427443 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:46.430018 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:46.430021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:46.430023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:46.430048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:46.431358 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:46.431844 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:46.431847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:46.431848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:46.431933 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:46.432224 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:46.432226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:46.432227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:46.432454 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:46.432532 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:46.432988 (+ 456us) server_negotiation.cc:299] Negotiation successful
0806 11:41:46.433084 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:41:46.433969 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:46.421656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43857 (local address 127.0.113.3:36903)
0806 11:41:46.422097 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:46.422136 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:46.423491 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:46.424041 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:46.424047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:46.424403 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:46.424853 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:46.424866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:46.425745 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:46.425752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:46.426883 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:46.426897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:46.427466 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:46.427469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:46.427673 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:46.428344 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:46.428390 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:46.429830 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:46.431478 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:46.431483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:46.431493 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:46.431737 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:46.432012 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:46.432014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:46.432015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:46.432149 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:46.432557 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:46.432562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:46.432788 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 11:41:46.432945 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 11:41:46.433200 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260806 11:41:46.435011 4588 heartbeater.cc:347] Connected to a master server at 127.0.113.62:43857
I20260806 11:41:46.435214 4588 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:46.435616 4588 heartbeater.cc:511] Master 127.0.113.62:43857 requested a full tablet report, sending...
I20260806 11:41:46.436882 3939 ts_manager.cc:194] Registered new tserver with Master: 0d8767e2dc04409098be7c0186225df8 (127.0.113.3:36513)
I20260806 11:41:46.437431 3939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:36903
I20260806 11:41:46.439327 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 11:41:46.453183 3989 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:41:46.451817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.0.113.62:43857)
0806 11:41:46.451946 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:46.451949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:46.452149 (+ 200us) server_negotiation.cc:407] Connection header received
0806 11:41:46.452244 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:46.452246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:46.452280 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:46.452409 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:46.453042 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48586: BlockingRecv error: recv got EOF from 127.0.0.1:48586 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:41:46.455904 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 4021
I20260806 11:41:46.471662 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 4216
I20260806 11:41:46.485630 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 4406
I20260806 11:41:46.498459 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 3894
2026-08-06T11:41:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2386 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:46 dist-test-slave-45dx krb5kdc[4604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:46 dist-test-slave-45dx krb5kdc[4604](info): set up 2 sockets
Aug 06 11:41:46 dist-test-slave-45dx krb5kdc[4604](info): commencing operation
krb5kdc: starting...
W20260806 11:41:47.725023 452 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:47 dist-test-slave-45dx krb5kdc[4604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:47.851311 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36871
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:47.960431 4619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:47.960656 4619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:47.960690 4619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:47.965099 4619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:47.965152 4619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:47.965170 4619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:47.965186 4619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:47.965200 4619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:47.970256 4619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:47.971150 4619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:47.972030 4619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 11:41:47.980517 4619 server_base.cc:1034] running on GCE node
W20260806 11:41:47.980446 4626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:47.980464 4625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:47.980446 4629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:47.981304 4619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:47.982321 4619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:47.983516 4619 hybrid_clock.cc:648] HybridClock initialized: now 1786016507983489 us; error 40 us; skew 500 ppm
Aug 06 11:41:47 dist-test-slave-45dx krb5kdc[4604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:47.986301 4619 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:47.987218 4619 webserver.cc:460] Webserver started at http://127.0.113.62:35703/ using document root <none> and password file <none>
I20260806 11:41:47.987785 4619 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:47.987877 4619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:47.988103 4619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:47.990155 4619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "073312385e344d8faa404da06dc7d72d"
format_stamp: "Formatted at 2026-08-06 11:41:47 on dist-test-slave-45dx"
server_key: "7a1f978c7a450211a663ae18aec09025"
server_key_iv: "1fdc8035dc4bd2eab504c0869a536b22"
server_key_version: "encryptionkey@0"
I20260806 11:41:47.990746 4619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "073312385e344d8faa404da06dc7d72d"
format_stamp: "Formatted at 2026-08-06 11:41:47 on dist-test-slave-45dx"
server_key: "7a1f978c7a450211a663ae18aec09025"
server_key_iv: "1fdc8035dc4bd2eab504c0869a536b22"
server_key_version: "encryptionkey@0"
I20260806 11:41:47.994354 4619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:41:47.996693 4638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:47.997884 4619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:41:47.998034 4619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "073312385e344d8faa404da06dc7d72d"
format_stamp: "Formatted at 2026-08-06 11:41:47 on dist-test-slave-45dx"
server_key: "7a1f978c7a450211a663ae18aec09025"
server_key_iv: "1fdc8035dc4bd2eab504c0869a536b22"
server_key_version: "encryptionkey@0"
I20260806 11:41:47.998143 4619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:48.012552 4619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:48.015739 4619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:48.015938 4619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:48.022872 4619 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:36871
I20260806 11:41:48.023177 4716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:36871 every 8 connection(s)
I20260806 11:41:48.023820 4619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:48.026528 4718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:48.027277 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 4619
I20260806 11:41:48.027397 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:48.027699 452 external_mini_cluster.cc:1428] Setting key 5035bda6506f283b8c49843284eaba0f
I20260806 11:41:48.032841 4718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d: Bootstrap starting.
I20260806 11:41:48.035570 4718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:48.036495 4718 log.cc:824] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d: Log is configured to *not* fsync() on all Append() calls
Aug 06 11:41:48 dist-test-slave-45dx krb5kdc[4604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:48.038383 4718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d: No bootstrap required, opened a new log
I20260806 11:41:48.041903 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:48.028952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.113.62:36871)
0806 11:41:48.029353 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:48.029374 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:48.029409 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:41:48.030033 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:48.030050 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:48.030349 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:48.030618 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:48.031564 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:48.032589 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:48.033731 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:48.034228 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:48.034349 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:48.037253 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:48.037258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:48.037272 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:48.037301 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:48.039066 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:48.039540 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:48.039545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:48.039550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:48.039663 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:48.039957 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:48.039962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:48.039963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:48.040415 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:48.040629 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:48.040904 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 11:41:48.041184 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":123,"threads_started":1}
I20260806 11:41:48.042019 4718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073312385e344d8faa404da06dc7d72d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36871 } }
I20260806 11:41:48.042294 4718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:48.042377 4718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 073312385e344d8faa404da06dc7d72d, State: Initialized, Role: FOLLOWER
I20260806 11:41:48.042810 4718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073312385e344d8faa404da06dc7d72d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36871 } }
I20260806 11:41:48.042956 4718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:48.043022 4718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:48.043118 4718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:48.044080 4718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073312385e344d8faa404da06dc7d72d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36871 } }
I20260806 11:41:48.044463 4718 leader_election.cc:302] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 073312385e344d8faa404da06dc7d72d; no voters:
I20260806 11:41:48.044793 4718 leader_election.cc:288] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:48.044867 4726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:48.045014 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:41:48.045082 4726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [term 1 LEADER]: Becoming Leader. State: Replica: 073312385e344d8faa404da06dc7d72d, State: Running, Role: LEADER
I20260806 11:41:48.045362 4726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073312385e344d8faa404da06dc7d72d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36871 } }
I20260806 11:41:48.046046 4718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:48.046509 4726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 073312385e344d8faa404da06dc7d72d. Latest consensus state: current_term: 1 leader_uuid: "073312385e344d8faa404da06dc7d72d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073312385e344d8faa404da06dc7d72d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36871 } } }
I20260806 11:41:48.046613 4726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:48.046943 4727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "073312385e344d8faa404da06dc7d72d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073312385e344d8faa404da06dc7d72d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36871 } } }
I20260806 11:41:48.047020 4731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:48.047055 4727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:48.050268 4731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:48.055672 4731 catalog_manager.cc:1347] Generated new cluster ID: 04521259f8ee4984ae46793d51623a8a
I20260806 11:41:48.055785 4731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:48.068976 4731 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:48.070158 4731 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:48.072360 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:48.066593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.0.113.62:36871)
0806 11:41:48.066710 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:48.066714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:48.067018 (+ 304us) server_negotiation.cc:407] Connection header received
0806 11:41:48.067148 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:48.067151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:48.067207 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:48.067304 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:48.067989 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:48.068733 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:48.069891 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:48.070279 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:48.070381 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:48.071323 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:48.071327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:48.071330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:48.071893 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:48.071926 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:48.071945 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:48.072034 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:48.072122 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 11:41:48.072196 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 11:41:48.075946 4731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 073312385e344d8faa404da06dc7d72d: Generated new TSK 0
I20260806 11:41:48.076622 4731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:41:48.083132 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 4619
2026-08-06T11:41:48Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1589 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:48 dist-test-slave-45dx krb5kdc[4769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:48 dist-test-slave-45dx krb5kdc[4769](info): set up 2 sockets
Aug 06 11:41:48 dist-test-slave-45dx krb5kdc[4769](info): commencing operation
krb5kdc: starting...
W20260806 11:41:49.332618 452 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:49 dist-test-slave-45dx krb5kdc[4769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:49.462867 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:40863
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:34409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:49.575603 4788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:49.575953 4788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:49.576040 4788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:49.580087 4788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:49.580179 4788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:49.580219 4788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:49.580252 4788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:49.580283 4788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:49.585445 4788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:49.586521 4788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:49.587549 4788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:49.596374 4795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:49.596340 4797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:49.596349 4794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:49.596657 4788 server_base.cc:1034] running on GCE node
I20260806 11:41:49.597224 4788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:49.598263 4788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:49.599435 4788 hybrid_clock.cc:648] HybridClock initialized: now 1786016509599405 us; error 42 us; skew 500 ppm
Aug 06 11:41:49 dist-test-slave-45dx krb5kdc[4769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:49.606849 4788 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:49.607820 4788 webserver.cc:460] Webserver started at http://127.0.113.62:34811/ using document root <none> and password file <none>
I20260806 11:41:49.608465 4788 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:49.608577 4788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:49.608783 4788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:49.610808 4788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "8cc3b3fd259d4713a181058d220b4c1c"
format_stamp: "Formatted at 2026-08-06 11:41:49 on dist-test-slave-45dx"
server_key: "dea46a5c545c06463dd02969cf52509f"
server_key_iv: "01f6b74b1d9c1dd8ac8e48beb05abe78"
server_key_version: "encryptionkey@0"
I20260806 11:41:49.611354 4788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "8cc3b3fd259d4713a181058d220b4c1c"
format_stamp: "Formatted at 2026-08-06 11:41:49 on dist-test-slave-45dx"
server_key: "dea46a5c545c06463dd02969cf52509f"
server_key_iv: "01f6b74b1d9c1dd8ac8e48beb05abe78"
server_key_version: "encryptionkey@0"
I20260806 11:41:49.615082 4788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 11:41:49.617449 4809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:49.618937 4788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:41:49.619064 4788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "8cc3b3fd259d4713a181058d220b4c1c"
format_stamp: "Formatted at 2026-08-06 11:41:49 on dist-test-slave-45dx"
server_key: "dea46a5c545c06463dd02969cf52509f"
server_key_iv: "01f6b74b1d9c1dd8ac8e48beb05abe78"
server_key_version: "encryptionkey@0"
I20260806 11:41:49.619165 4788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:49.645732 4788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:49.649256 4788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:49.649490 4788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:49.656867 4788 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:40863
I20260806 11:41:49.657236 4884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:40863 every 8 connection(s)
I20260806 11:41:49.657846 4788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:49.659512 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 4788
I20260806 11:41:49.659617 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:49.659914 452 external_mini_cluster.cc:1428] Setting key f48e40767e762c6c17fa0343e5787ab5
I20260806 11:41:49.660938 4885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:49.667511 4885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c: Bootstrap starting.
Aug 06 11:41:49 dist-test-slave-45dx krb5kdc[4769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:49.670095 4885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:49.671098 4885 log.cc:824] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:49.673573 4885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c: No bootstrap required, opened a new log
I20260806 11:41:49.674929 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:49.661358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.0.113.62:40863)
0806 11:41:49.661748 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:49.661763 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:49.661808 (+ 45us) server_negotiation.cc:407] Connection header received
0806 11:41:49.662475 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:49.662491 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:49.662796 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:49.663076 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:49.664045 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:49.665095 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:49.666342 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:49.666799 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:49.666918 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:49.669705 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:49.669710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:49.669723 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:49.669751 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:49.671772 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:49.672231 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:49.672238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:49.672243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:49.672429 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:49.672708 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:49.672711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:49.672713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:49.673236 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:49.673440 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:49.673696 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 11:41:49.674006 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260806 11:41:49.677362 4885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc3b3fd259d4713a181058d220b4c1c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40863 } }
I20260806 11:41:49.677690 4885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:49.677809 4885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc3b3fd259d4713a181058d220b4c1c, State: Initialized, Role: FOLLOWER
I20260806 11:41:49.678203 4885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc3b3fd259d4713a181058d220b4c1c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40863 } }
I20260806 11:41:49.678320 4885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:49.678359 4885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:49.678426 4885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:49.679275 4885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc3b3fd259d4713a181058d220b4c1c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40863 } }
I20260806 11:41:49.679562 4885 leader_election.cc:302] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cc3b3fd259d4713a181058d220b4c1c; no voters:
I20260806 11:41:49.679817 4885 leader_election.cc:288] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:49.680080 4892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:49.680454 4892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [term 1 LEADER]: Becoming Leader. State: Replica: 8cc3b3fd259d4713a181058d220b4c1c, State: Running, Role: LEADER
I20260806 11:41:49.680778 4892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc3b3fd259d4713a181058d220b4c1c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40863 } }
I20260806 11:41:49.681263 4885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:49.682952 4893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cc3b3fd259d4713a181058d220b4c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc3b3fd259d4713a181058d220b4c1c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40863 } } }
I20260806 11:41:49.683354 4893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:49.683238 4894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cc3b3fd259d4713a181058d220b4c1c. Latest consensus state: current_term: 1 leader_uuid: "8cc3b3fd259d4713a181058d220b4c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc3b3fd259d4713a181058d220b4c1c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40863 } } }
I20260806 11:41:49.683689 4894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:49.683771 4911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:49.687260 4911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:49.693120 4911 catalog_manager.cc:1347] Generated new cluster ID: e855f75b6e3b4b1598838811cb9bb3f0
I20260806 11:41:49.693193 4911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:49.723042 4911 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:49.724251 4911 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:49.729754 4911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cc3b3fd259d4713a181058d220b4c1c: Generated new TSK 0
I20260806 11:41:49.730587 4911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:41:49.763491 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40863
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:34409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:49.875209 4921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:49.875468 4921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:49.875509 4921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:49.879443 4921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:49.879496 4921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:49.879551 4921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:49.884873 4921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:49.885814 4921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:49.886754 4921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:49.896337 4926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:49.896344 4929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:49.896354 4927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:49.896693 4921 server_base.cc:1034] running on GCE node
I20260806 11:41:49.897125 4921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:49.897740 4921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:49.898928 4921 hybrid_clock.cc:648] HybridClock initialized: now 1786016509898912 us; error 25 us; skew 500 ppm
Aug 06 11:41:49 dist-test-slave-45dx krb5kdc[4769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:49.901959 4921 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:41:49.902832 4921 webserver.cc:460] Webserver started at http://127.0.113.1:34905/ using document root <none> and password file <none>
I20260806 11:41:49.903390 4921 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:49.903483 4921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:49.903713 4921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:49.905798 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "6dbe2ab6ffbd45c4ac98dbd1996d690f"
format_stamp: "Formatted at 2026-08-06 11:41:49 on dist-test-slave-45dx"
server_key: "0feb27651657155c53278f2a7c973922"
server_key_iv: "5924c2352aa86b75db4130b66b06ac85"
server_key_version: "encryptionkey@0"
I20260806 11:41:49.906376 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "6dbe2ab6ffbd45c4ac98dbd1996d690f"
format_stamp: "Formatted at 2026-08-06 11:41:49 on dist-test-slave-45dx"
server_key: "0feb27651657155c53278f2a7c973922"
server_key_iv: "5924c2352aa86b75db4130b66b06ac85"
server_key_version: "encryptionkey@0"
I20260806 11:41:49.910065 4921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:41:49.912554 4940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:49.913733 4921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:49.913862 4921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "6dbe2ab6ffbd45c4ac98dbd1996d690f"
format_stamp: "Formatted at 2026-08-06 11:41:49 on dist-test-slave-45dx"
server_key: "0feb27651657155c53278f2a7c973922"
server_key_iv: "5924c2352aa86b75db4130b66b06ac85"
server_key_version: "encryptionkey@0"
I20260806 11:41:49.913971 4921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:49.933888 4921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:49.937140 4921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:49.937335 4921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:49.937903 4921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:49.938684 4921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:49.938751 4921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:49.938818 4921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:49.938865 4921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:49.947964 4921 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:44123
I20260806 11:41:49.947991 5097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:44123 every 8 connection(s)
I20260806 11:41:49.948846 4921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:49.950044 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 4921
I20260806 11:41:49.950184 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:49.950507 452 external_mini_cluster.cc:1428] Setting key 25c10d4f3c7d3f76790da50056bd1308
Aug 06 11:41:49 dist-test-slave-45dx krb5kdc[4769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:41:49.962821 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:49.950664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:36653 (local address 127.0.113.62:40863)
0806 11:41:49.950819 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:49.950824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:49.951737 (+ 913us) server_negotiation.cc:407] Connection header received
0806 11:41:49.953217 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:49.953220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:49.953272 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:49.953343 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:49.954663 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:49.955184 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:49.956341 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:49.956632 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:49.956769 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:49.959146 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:49.959150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:49.959153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:49.959192 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:49.960714 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:49.961306 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:49.961309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:49.961310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:49.961392 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:49.961743 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:49.961745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:49.961746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:49.962026 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:49.962108 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:49.962555 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 11:41:49.962653 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:41:49.963635 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:49.951095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40863 (local address 127.0.113.1:36653)
0806 11:41:49.951604 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:49.951643 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:49.953034 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:49.953497 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:49.953503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:49.953866 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:49.954506 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:49.954526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:49.955325 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:49.955328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:49.956191 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:49.956200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:49.956822 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:49.956826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:49.956992 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:49.957548 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:49.957565 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:49.958988 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:49.960874 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:49.960880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:49.960894 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:49.961216 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:49.961505 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:49.961508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:49.961509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:49.961648 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:49.962123 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:49.962128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:49.962344 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 11:41:49.962504 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 11:41:49.962738 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":177,"threads_started":1}
I20260806 11:41:49.964963 5099 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40863
I20260806 11:41:49.965215 5099 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:49.965850 5099 heartbeater.cc:511] Master 127.0.113.62:40863 requested a full tablet report, sending...
I20260806 11:41:49.967468 4831 ts_manager.cc:194] Registered new tserver with Master: 6dbe2ab6ffbd45c4ac98dbd1996d690f (127.0.113.1:44123)
I20260806 11:41:49.968698 4831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:36653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:41:49.979576 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40863
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:34409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:50.091841 5106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:50.092075 5106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:50.092115 5106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:50.096107 5106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:50.096160 5106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:50.096213 5106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:41:50.101233 5106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:50.102113 5106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:50.103075 5106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:50.112558 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:50.112600 5120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:50.112746 5106 server_base.cc:1034] running on GCE node
W20260806 11:41:50.112820 5123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:50.113220 5106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:50.113754 5106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:50.114948 5106 hybrid_clock.cc:648] HybridClock initialized: now 1786016510114928 us; error 25 us; skew 500 ppm
Aug 06 11:41:50 dist-test-slave-45dx krb5kdc[4769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:50.118320 5106 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:41:50.119343 5106 webserver.cc:460] Webserver started at http://127.0.113.2:46431/ using document root <none> and password file <none>
I20260806 11:41:50.120026 5106 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:50.120121 5106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:50.120368 5106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:50.122542 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "a1d284f8221045ffa3f299450afd75fb"
format_stamp: "Formatted at 2026-08-06 11:41:50 on dist-test-slave-45dx"
server_key: "9e9e8bed73f5ea5d608502f99989f07c"
server_key_iv: "fe97c60b67211c19f8ad79a0a6192025"
server_key_version: "encryptionkey@0"
I20260806 11:41:50.123104 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "a1d284f8221045ffa3f299450afd75fb"
format_stamp: "Formatted at 2026-08-06 11:41:50 on dist-test-slave-45dx"
server_key: "9e9e8bed73f5ea5d608502f99989f07c"
server_key_iv: "fe97c60b67211c19f8ad79a0a6192025"
server_key_version: "encryptionkey@0"
I20260806 11:41:50.126840 5106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 11:41:50.129284 5134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:50.130486 5106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:50.130581 5106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "a1d284f8221045ffa3f299450afd75fb"
format_stamp: "Formatted at 2026-08-06 11:41:50 on dist-test-slave-45dx"
server_key: "9e9e8bed73f5ea5d608502f99989f07c"
server_key_iv: "fe97c60b67211c19f8ad79a0a6192025"
server_key_version: "encryptionkey@0"
I20260806 11:41:50.130666 5106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:50.146363 5106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:50.149085 5106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:50.149240 5106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:50.149730 5106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:50.150517 5106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:50.150565 5106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:50.150604 5106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:50.150625 5106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:50.159577 5106 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:35253
I20260806 11:41:50.159601 5307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:35253 every 8 connection(s)
I20260806 11:41:50.160472 5106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:41:50.166244 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 5106
I20260806 11:41:50.166395 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:41:50.166715 452 external_mini_cluster.cc:1428] Setting key b4b4a1c759dfc0774aaf28d3b3a3da56
Aug 06 11:41:50 dist-test-slave-45dx krb5kdc[4769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:50.174714 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:50.162195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:48249 (local address 127.0.113.62:40863)
0806 11:41:50.162337 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:50.162341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:50.163060 (+ 719us) server_negotiation.cc:407] Connection header received
0806 11:41:50.164459 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:50.164462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:50.164523 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:50.164628 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:50.165751 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.166262 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.167835 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.168096 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.168206 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:50.170726 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:50.170729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:50.170731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:50.170756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:50.172250 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:50.173016 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:50.173020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:50.173021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:50.173105 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:50.173542 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:50.173544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:50.173545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:50.173726 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:50.173807 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:50.174487 (+ 680us) server_negotiation.cc:299] Negotiation successful
0806 11:41:50.174592 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:41:50.175994 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:50.162489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40863 (local address 127.0.113.2:48249)
0806 11:41:50.162909 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:50.162949 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:50.164218 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:50.164791 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:50.164797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:50.165132 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:50.165591 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:50.165604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.166431 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.166435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:50.167709 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:50.167720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.168255 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.168259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:50.168477 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:50.169134 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:50.169150 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:50.170561 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:50.172466 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:50.172474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:50.172492 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:50.172910 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:50.173208 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:50.173215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:50.173220 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:50.173441 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:50.173839 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:50.173848 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:50.174194 (+ 346us) client_negotiation.cc:769] Sending connection context
0806 11:41:50.174435 (+ 241us) client_negotiation.cc:240] Negotiation successful
0806 11:41:50.174810 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260806 11:41:50.177274 5308 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40863
I20260806 11:41:50.177534 5308 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:50.178020 5308 heartbeater.cc:511] Master 127.0.113.62:40863 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:41:50.179104 4831 ts_manager.cc:194] Registered new tserver with Master: a1d284f8221045ffa3f299450afd75fb (127.0.113.2:35253)
I20260806 11:41:50.179774 4831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:48249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:41:50.197469 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40863
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:34409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:50.311139 5313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:50.311487 5313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:50.311563 5313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:50.315557 5313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:50.315610 5313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:50.315666 5313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:41:50.320860 5313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:50.321718 5313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:50.322782 5313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:50.332135 5320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:50.332216 5319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:50.332180 5322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:50.332499 5313 server_base.cc:1034] running on GCE node
I20260806 11:41:50.332875 5313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:50.333503 5313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:50.334695 5313 hybrid_clock.cc:648] HybridClock initialized: now 1786016510334654 us; error 52 us; skew 500 ppm
Aug 06 11:41:50 dist-test-slave-45dx krb5kdc[4769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:50.337540 5313 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:41:50.338413 5313 webserver.cc:460] Webserver started at http://127.0.113.3:40325/ using document root <none> and password file <none>
I20260806 11:41:50.338907 5313 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:50.338965 5313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:50.339124 5313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:50.341068 5313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "e259d161675f4f7494a7d529dfa52ab1"
format_stamp: "Formatted at 2026-08-06 11:41:50 on dist-test-slave-45dx"
server_key: "00668e55c6098bb96eddba3bc2137bd0"
server_key_iv: "8e978c3f3457bb9c90a9106073b8d17c"
server_key_version: "encryptionkey@0"
I20260806 11:41:50.341599 5313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "e259d161675f4f7494a7d529dfa52ab1"
format_stamp: "Formatted at 2026-08-06 11:41:50 on dist-test-slave-45dx"
server_key: "00668e55c6098bb96eddba3bc2137bd0"
server_key_iv: "8e978c3f3457bb9c90a9106073b8d17c"
server_key_version: "encryptionkey@0"
I20260806 11:41:50.345206 5313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 11:41:50.347468 5334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:50.348693 5313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:41:50.348790 5313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "e259d161675f4f7494a7d529dfa52ab1"
format_stamp: "Formatted at 2026-08-06 11:41:50 on dist-test-slave-45dx"
server_key: "00668e55c6098bb96eddba3bc2137bd0"
server_key_iv: "8e978c3f3457bb9c90a9106073b8d17c"
server_key_version: "encryptionkey@0"
I20260806 11:41:50.348870 5313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:50.365561 5313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:50.368795 5313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:50.368942 5313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:50.369412 5313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:50.370155 5313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:50.370201 5313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:50.370239 5313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:50.370260 5313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:50.379143 5313 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:44553
I20260806 11:41:50.379169 5490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:44553 every 8 connection(s)
I20260806 11:41:50.380010 5313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:41:50.384156 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 5313
I20260806 11:41:50.384275 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:41:50.384670 452 external_mini_cluster.cc:1428] Setting key 2a4ca47fec23a19344f79011e83951fa
Aug 06 11:41:50 dist-test-slave-45dx krb5kdc[4769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:50.394053 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:50.381749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:43651 (local address 127.0.113.62:40863)
0806 11:41:50.381965 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:50.381968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:50.382788 (+ 820us) server_negotiation.cc:407] Connection header received
0806 11:41:50.384403 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:50.384407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:50.384465 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:50.384537 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:50.386014 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.386526 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.387666 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.387891 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.388003 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:50.390570 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:50.390573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:50.390575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:50.390601 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:50.392025 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:50.392632 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:50.392635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:50.392636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:50.392716 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:50.393034 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:50.393036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:50.393037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:50.393210 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:50.393285 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:50.393830 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 11:41:50.393935 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260806 11:41:50.394923 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:50.382173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40863 (local address 127.0.113.3:43651)
0806 11:41:50.382612 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:50.382653 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:50.384148 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:50.384738 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:50.384747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:50.385216 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:50.385848 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:50.385866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.386653 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.386657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:50.387535 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:50.387544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.388043 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.388048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:50.388214 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:50.388845 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:50.388861 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:50.390424 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:50.392163 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:50.392168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:50.392179 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:50.392542 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:50.392803 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:50.392806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:50.392807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:50.392944 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:50.393336 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:50.393341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:50.393600 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 11:41:50.393759 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 11:41:50.394014 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":130,"threads_started":1}
I20260806 11:41:50.396025 5491 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40863
I20260806 11:41:50.396265 5491 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:50.396768 5491 heartbeater.cc:511] Master 127.0.113.62:40863 requested a full tablet report, sending...
I20260806 11:41:50.397801 4831 ts_manager.cc:194] Registered new tserver with Master: e259d161675f4f7494a7d529dfa52ab1 (127.0.113.3:44553)
I20260806 11:41:50.398465 4831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:43651
I20260806 11:41:50.398515 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 11:41:50 dist-test-slave-45dx krb5kdc[4769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016510, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 11:41:50 dist-test-slave-45dx krb5kdc[4769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016510, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:50.425364 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:50.417055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.0.113.62:40863)
0806 11:41:50.417204 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:50.417208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:50.417380 (+ 172us) server_negotiation.cc:407] Connection header received
0806 11:41:50.417490 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:50.417493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:50.417533 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:50.417605 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:50.418377 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.418871 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.419653 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:50.419852 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:50.419941 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:50.422226 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:50.422230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:50.422235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:50.422263 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:50.423904 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:50.424254 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:50.424256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:50.424257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:50.424410 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:50.424642 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:50.424644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:50.424645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:50.424845 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:50.424975 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:50.425167 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 11:41:50.425254 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 11:41:50.425724 4831 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48210
I20260806 11:41:50.426540 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 4921
I20260806 11:41:50.437939 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 5106
I20260806 11:41:50.450804 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 5313
I20260806 11:41:50.460011 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 4788
2026-08-06T11:41:50Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2375 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:50 dist-test-slave-45dx krb5kdc[5508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:50 dist-test-slave-45dx krb5kdc[5508](info): set up 2 sockets
Aug 06 11:41:50 dist-test-slave-45dx krb5kdc[5508](info): commencing operation
krb5kdc: starting...
W20260806 11:41:51.666095 452 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:51 dist-test-slave-45dx krb5kdc[5508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:51.790941 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:42691
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:42691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:51.900249 5533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:51.900506 5533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:51.900542 5533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:51.904495 5533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:51.904546 5533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:51.904564 5533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:51.904613 5533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:51.904642 5533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:51.909785 5533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:42691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:42691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:51.910719 5533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:51.911669 5533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:51.919661 5539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:51.919746 5540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:51.919661 5543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:51.920037 5533 server_base.cc:1034] running on GCE node
I20260806 11:41:51.920686 5533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:51.921686 5533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:51.922878 5533 hybrid_clock.cc:648] HybridClock initialized: now 1786016511922864 us; error 32 us; skew 500 ppm
Aug 06 11:41:51 dist-test-slave-45dx krb5kdc[5508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:51.925747 5533 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:51.926579 5533 webserver.cc:460] Webserver started at http://127.0.113.62:44193/ using document root <none> and password file <none>
I20260806 11:41:51.927145 5533 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:51.927241 5533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:51.927471 5533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:51.929472 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "0b94ab5f5c464dccb0d0697613e3092b"
format_stamp: "Formatted at 2026-08-06 11:41:51 on dist-test-slave-45dx"
server_key: "d95046126260e5090222f0ff761cc4b8"
server_key_iv: "641abfd0a2692dd2944385a2257ba1a0"
server_key_version: "encryptionkey@0"
I20260806 11:41:51.930063 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "0b94ab5f5c464dccb0d0697613e3092b"
format_stamp: "Formatted at 2026-08-06 11:41:51 on dist-test-slave-45dx"
server_key: "d95046126260e5090222f0ff761cc4b8"
server_key_iv: "641abfd0a2692dd2944385a2257ba1a0"
server_key_version: "encryptionkey@0"
I20260806 11:41:51.933669 5533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:41:51.936054 5556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:51.937331 5533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:51.937480 5533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "0b94ab5f5c464dccb0d0697613e3092b"
format_stamp: "Formatted at 2026-08-06 11:41:51 on dist-test-slave-45dx"
server_key: "d95046126260e5090222f0ff761cc4b8"
server_key_iv: "641abfd0a2692dd2944385a2257ba1a0"
server_key_version: "encryptionkey@0"
I20260806 11:41:51.937597 5533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:51.956538 5533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:51.959697 5533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:51.959908 5533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:51.966746 5533 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:42691
I20260806 11:41:51.966773 5633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:42691 every 8 connection(s)
I20260806 11:41:51.967760 5533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:51.970508 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:51.975673 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b: Bootstrap starting.
I20260806 11:41:51.977377 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 5533
I20260806 11:41:51.977514 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:51.977825 452 external_mini_cluster.cc:1428] Setting key f37a6c38484acf232808dad55c36ee92
I20260806 11:41:51.978499 5634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:51.979391 5634 log.cc:824] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:51.981336 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b: No bootstrap required, opened a new log
I20260806 11:41:51.984687 5634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94ab5f5c464dccb0d0697613e3092b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42691 } }
I20260806 11:41:51.984979 5634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:51.985065 5634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b94ab5f5c464dccb0d0697613e3092b, State: Initialized, Role: FOLLOWER
I20260806 11:41:51.985502 5634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94ab5f5c464dccb0d0697613e3092b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42691 } }
I20260806 11:41:51.985672 5634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:51.985745 5634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:51.985845 5634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [term 0 FOLLOWER]: Advancing to term 1
Aug 06 11:41:51 dist-test-slave-45dx krb5kdc[5508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:51.986754 5634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94ab5f5c464dccb0d0697613e3092b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42691 } }
I20260806 11:41:51.987115 5634 leader_election.cc:302] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b94ab5f5c464dccb0d0697613e3092b; no voters:
I20260806 11:41:51.987483 5634 leader_election.cc:288] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:51.987519 5641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:51.987710 5641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [term 1 LEADER]: Becoming Leader. State: Replica: 0b94ab5f5c464dccb0d0697613e3092b, State: Running, Role: LEADER
I20260806 11:41:51.988111 5641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94ab5f5c464dccb0d0697613e3092b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42691 } }
I20260806 11:41:51.988739 5634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:51.989804 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b94ab5f5c464dccb0d0697613e3092b. Latest consensus state: current_term: 1 leader_uuid: "0b94ab5f5c464dccb0d0697613e3092b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94ab5f5c464dccb0d0697613e3092b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42691 } } }
I20260806 11:41:51.989945 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:51.990096 5644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b94ab5f5c464dccb0d0697613e3092b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94ab5f5c464dccb0d0697613e3092b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42691 } } }
I20260806 11:41:51.990325 5644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:51.990577 5653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:51.992373 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:51.978942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.0.113.62:42691)
0806 11:41:51.979361 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:51.979375 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:51.979425 (+ 50us) server_negotiation.cc:407] Connection header received
0806 11:41:51.980248 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:51.980268 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:51.980635 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:51.980956 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:51.981843 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:51.982781 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:51.983634 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:51.984082 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:51.984247 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:51.986935 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:51.986944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:51.986960 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:51.986993 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:51.988952 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:51.989527 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:51.989535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:51.989543 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:51.989707 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:51.990100 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:51.990105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:51.990110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:51.990637 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:51.990833 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:51.991203 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 11:41:51.991480 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
I20260806 11:41:51.993609 5653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:51.999055 5653 catalog_manager.cc:1347] Generated new cluster ID: 78faa265fc3a4814bef7faa83e4fe58a
I20260806 11:41:51.999375 5653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:52.006388 5653 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:52.007364 5653 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:52.012450 5653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b94ab5f5c464dccb0d0697613e3092b: Generated new TSK 0
I20260806 11:41:52.013131 5653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:41:52.048161 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42691
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:52.161100 5674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:52.161329 5674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:52.161367 5674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:52.165346 5674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:52.165400 5674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:52.165454 5674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:52.170468 5674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:52.171470 5674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:52.172566 5674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:52.182044 5686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:52.182137 5674 server_base.cc:1034] running on GCE node
W20260806 11:41:52.182058 5683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:52.182058 5681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:52.182739 5674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:52.183367 5674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:52.184561 5674 hybrid_clock.cc:648] HybridClock initialized: now 1786016512184541 us; error 34 us; skew 500 ppm
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[5508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:52.187441 5674 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:41:52.188386 5674 webserver.cc:460] Webserver started at http://127.0.113.1:36343/ using document root <none> and password file <none>
I20260806 11:41:52.188993 5674 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:52.189088 5674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:52.189303 5674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:52.191390 5674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "4471702d116f48dfaef13e6ab72fb6b5"
format_stamp: "Formatted at 2026-08-06 11:41:52 on dist-test-slave-45dx"
server_key: "c00402b549412521b4b350ba59775c4f"
server_key_iv: "fa68e6864c4a77b489582a987ff60063"
server_key_version: "encryptionkey@0"
I20260806 11:41:52.191982 5674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "4471702d116f48dfaef13e6ab72fb6b5"
format_stamp: "Formatted at 2026-08-06 11:41:52 on dist-test-slave-45dx"
server_key: "c00402b549412521b4b350ba59775c4f"
server_key_iv: "fa68e6864c4a77b489582a987ff60063"
server_key_version: "encryptionkey@0"
I20260806 11:41:52.195691 5674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:41:52.198082 5695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:52.199230 5674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 11:41:52.199365 5674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "4471702d116f48dfaef13e6ab72fb6b5"
format_stamp: "Formatted at 2026-08-06 11:41:52 on dist-test-slave-45dx"
server_key: "c00402b549412521b4b350ba59775c4f"
server_key_iv: "fa68e6864c4a77b489582a987ff60063"
server_key_version: "encryptionkey@0"
I20260806 11:41:52.199487 5674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:52.224241 5674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:52.227654 5674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:52.227866 5674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:52.228514 5674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:52.229390 5674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:52.229471 5674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:52.229542 5674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:52.229592 5674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:52.239070 5674 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:41703
I20260806 11:41:52.239099 5846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:41703 every 8 connection(s)
I20260806 11:41:52.239967 5674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:52.245283 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 5674
I20260806 11:41:52.245424 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:52.245998 452 external_mini_cluster.cc:1428] Setting key ea2e289f636b0f0b9e997a90735d7665
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[5508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:52.254649 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:52.241842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:51129 (local address 127.0.113.62:42691)
0806 11:41:52.241992 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:52.241997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:52.242853 (+ 856us) server_negotiation.cc:407] Connection header received
0806 11:41:52.244210 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:52.244213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:52.244262 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:52.244389 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:52.245638 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.246429 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.247709 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.248097 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.248228 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:52.250837 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:52.250840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:52.250842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:52.250866 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:52.252555 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.253068 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.253071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.253073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.253166 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.253519 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.253523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.253525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.253715 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:52.253841 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:52.254318 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 11:41:52.254461 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 11:41:52.255414 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:52.242256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:42691 (local address 127.0.113.1:51129)
0806 11:41:52.242700 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:52.242743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:52.244021 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:52.244540 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:52.244547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:52.244890 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:52.245372 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:52.245418 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.246564 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.246570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:52.247592 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:52.247602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.248261 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.248264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:52.248454 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:52.249045 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:52.249060 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:52.250662 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:52.252673 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:52.252679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:52.252693 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:52.252962 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:52.253259 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:52.253263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:52.253265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:52.253422 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:52.253841 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:52.253849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:52.254081 (+ 232us) client_negotiation.cc:769] Sending connection context
0806 11:41:52.254270 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 11:41:52.254600 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:41:52.256803 5847 heartbeater.cc:347] Connected to a master server at 127.0.113.62:42691
I20260806 11:41:52.257046 5847 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:52.257566 5847 heartbeater.cc:511] Master 127.0.113.62:42691 requested a full tablet report, sending...
I20260806 11:41:52.259133 5580 ts_manager.cc:194] Registered new tserver with Master: 4471702d116f48dfaef13e6ab72fb6b5 (127.0.113.1:41703)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:41:52.260764 5580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:51129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:41:52.276889 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42691
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:52.383785 5852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:52.384053 5852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:52.384128 5852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:52.388285 5852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:52.388379 5852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:52.388507 5852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:41:52.393960 5852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:52.394913 5852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:52.395848 5852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:52.405360 5862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:52.405391 5860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:52.405427 5852 server_base.cc:1034] running on GCE node
W20260806 11:41:52.405416 5859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:52.406033 5852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:52.406646 5852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:52.407841 5852 hybrid_clock.cc:648] HybridClock initialized: now 1786016512407815 us; error 47 us; skew 500 ppm
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[5508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:52.410668 5852 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:41:52.411451 5852 webserver.cc:460] Webserver started at http://127.0.113.2:44433/ using document root <none> and password file <none>
I20260806 11:41:52.411935 5852 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:52.411991 5852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:52.412179 5852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:52.414178 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "0ee4c661a7664c2abbf261b2126e8a72"
format_stamp: "Formatted at 2026-08-06 11:41:52 on dist-test-slave-45dx"
server_key: "915d9258e451c7a2b91c38a0da9b098a"
server_key_iv: "ca73d12d2a11b62ebee2d575c8e47c68"
server_key_version: "encryptionkey@0"
I20260806 11:41:52.414671 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "0ee4c661a7664c2abbf261b2126e8a72"
format_stamp: "Formatted at 2026-08-06 11:41:52 on dist-test-slave-45dx"
server_key: "915d9258e451c7a2b91c38a0da9b098a"
server_key_iv: "ca73d12d2a11b62ebee2d575c8e47c68"
server_key_version: "encryptionkey@0"
I20260806 11:41:52.418119 5852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:41:52.420291 5876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:52.421432 5852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:41:52.421555 5852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "0ee4c661a7664c2abbf261b2126e8a72"
format_stamp: "Formatted at 2026-08-06 11:41:52 on dist-test-slave-45dx"
server_key: "915d9258e451c7a2b91c38a0da9b098a"
server_key_iv: "ca73d12d2a11b62ebee2d575c8e47c68"
server_key_version: "encryptionkey@0"
I20260806 11:41:52.421653 5852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:52.431945 5852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:52.434556 5852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:52.434738 5852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:52.435293 5852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:52.436142 5852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:52.436189 5852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:52.436254 5852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:52.436295 5852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:52.445506 5852 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:41703
I20260806 11:41:52.445531 6017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:41703 every 8 connection(s)
I20260806 11:41:52.446281 5852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:41:52.452955 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 5852
I20260806 11:41:52.453081 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:41:52.453409 452 external_mini_cluster.cc:1428] Setting key bb77b872ce7bed889336128af0b123a0
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[5508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:52.461120 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:52.447981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:36309 (local address 127.0.113.62:42691)
0806 11:41:52.448107 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:52.448110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:52.448926 (+ 816us) server_negotiation.cc:407] Connection header received
0806 11:41:52.450241 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:52.450245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:52.450302 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:52.450378 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:52.451449 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.452012 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.453778 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.454091 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.454211 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:52.456924 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:52.456948 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:52.456953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:52.456987 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:52.458743 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.459473 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.459476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.459477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.459584 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.459990 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.459993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.459994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.460153 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:52.460223 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:52.460908 (+ 685us) server_negotiation.cc:299] Negotiation successful
0806 11:41:52.461006 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:41:52.462345 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:52.448268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:42691 (local address 127.0.113.2:36309)
0806 11:41:52.448787 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:52.448826 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:52.450043 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:52.450538 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:52.450544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:52.450855 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:52.451302 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:52.451314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.452350 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.452356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:52.453652 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:52.453665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.454248 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.454253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:52.454424 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:52.454978 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:52.454994 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:52.456716 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:52.458875 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:52.458884 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:52.458899 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:52.459320 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:52.459680 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:52.459686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:52.459691 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:52.459886 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:52.460267 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:52.460283 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:52.460635 (+ 352us) client_negotiation.cc:769] Sending connection context
0806 11:41:52.460850 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 11:41:52.461186 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":151,"threads_started":1}
I20260806 11:41:52.463536 6019 heartbeater.cc:347] Connected to a master server at 127.0.113.62:42691
I20260806 11:41:52.463760 6019 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:52.464210 6019 heartbeater.cc:511] Master 127.0.113.62:42691 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:41:52.465303 5580 ts_manager.cc:194] Registered new tserver with Master: 0ee4c661a7664c2abbf261b2126e8a72 (127.0.113.2:41703)
I20260806 11:41:52.465845 5580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:36309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:41:52.482853 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42691
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:52.594228 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:52.594494 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:52.594538 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:52.598512 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:52.598565 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:52.598621 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:41:52.603860 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:52.604803 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:52.605728 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:52.615417 6037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:52.615425 6041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:52.615433 6029 server_base.cc:1034] running on GCE node
W20260806 11:41:52.615433 6038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:52.616071 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:52.616736 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:52.617899 6029 hybrid_clock.cc:648] HybridClock initialized: now 1786016512617868 us; error 45 us; skew 500 ppm
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[5508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:52.620564 6029 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:41:52.621564 6029 webserver.cc:460] Webserver started at http://127.0.113.3:46539/ using document root <none> and password file <none>
I20260806 11:41:52.622141 6029 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:52.622241 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:52.622478 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:52.624522 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "e162c6cecf314d40af0ea700baebc779"
format_stamp: "Formatted at 2026-08-06 11:41:52 on dist-test-slave-45dx"
server_key: "de2af384de765acaf049ac220c904670"
server_key_iv: "8965718989a04850f808be27ec8209e4"
server_key_version: "encryptionkey@0"
I20260806 11:41:52.625095 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "e162c6cecf314d40af0ea700baebc779"
format_stamp: "Formatted at 2026-08-06 11:41:52 on dist-test-slave-45dx"
server_key: "de2af384de765acaf049ac220c904670"
server_key_iv: "8965718989a04850f808be27ec8209e4"
server_key_version: "encryptionkey@0"
I20260806 11:41:52.628938 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 11:41:52.631258 6052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:52.632494 6029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:52.632623 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "e162c6cecf314d40af0ea700baebc779"
format_stamp: "Formatted at 2026-08-06 11:41:52 on dist-test-slave-45dx"
server_key: "de2af384de765acaf049ac220c904670"
server_key_iv: "8965718989a04850f808be27ec8209e4"
server_key_version: "encryptionkey@0"
I20260806 11:41:52.632730 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:52.655738 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:52.658476 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:52.658674 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:52.659236 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:52.660189 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:52.660261 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:52.660379 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:52.660431 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:52.670096 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:43305
I20260806 11:41:52.670125 6210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:43305 every 8 connection(s)
I20260806 11:41:52.671056 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:41:52.679028 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 6029
I20260806 11:41:52.679157 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:41:52.679515 452 external_mini_cluster.cc:1428] Setting key f400d9aef45c70e0da63860826ba6c5a
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[5508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:52.684691 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:52.672942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:47177 (local address 127.0.113.62:42691)
0806 11:41:52.673098 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:52.673103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:52.673991 (+ 888us) server_negotiation.cc:407] Connection header received
0806 11:41:52.675374 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:52.675377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:52.675427 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:52.675509 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:52.676757 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.677302 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.678412 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.678624 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.678703 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:52.681552 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:52.681555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:52.681557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:52.681581 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:52.682885 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.683355 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.683358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.683359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.683442 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.683741 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.683743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.683744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.683915 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:52.683991 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:52.684479 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 11:41:52.684573 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 11:41:52.685505 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:52.673377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:42691 (local address 127.0.113.3:47177)
0806 11:41:52.673830 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:52.673876 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:52.675151 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:52.675673 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:52.675680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:52.676042 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:52.676607 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:52.676623 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.677426 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.677428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:52.678300 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:52.678309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.678753 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.678756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:52.678893 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:52.679584 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:52.679612 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:52.681399 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:52.682990 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:52.682994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:52.683005 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:52.683268 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:52.683522 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:52.683525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:52.683526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:52.683662 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:52.684000 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:52.684005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:52.684219 (+ 214us) client_negotiation.cc:769] Sending connection context
0806 11:41:52.684431 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 11:41:52.684686 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260806 11:41:52.686584 6211 heartbeater.cc:347] Connected to a master server at 127.0.113.62:42691
I20260806 11:41:52.686820 6211 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:52.687319 6211 heartbeater.cc:511] Master 127.0.113.62:42691 requested a full tablet report, sending...
I20260806 11:41:52.688433 5580 ts_manager.cc:194] Registered new tserver with Master: e162c6cecf314d40af0ea700baebc779 (127.0.113.3:43305)
I20260806 11:41:52.688980 5580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:47177
I20260806 11:41:52.693514 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[5508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[5508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:41:52.720157 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:52.711149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.0.113.1:41703)
0806 11:41:52.711423 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:52.711448 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:52.711471 (+ 23us) server_negotiation.cc:407] Connection header received
0806 11:41:52.711522 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:52.711527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:52.711705 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:52.711832 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:52.712557 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.713351 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.714156 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.714511 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.714593 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:52.716928 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:52.716937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:52.716950 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:52.716974 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:52.718516 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.718914 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.718919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.718920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.719001 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.719228 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.719230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.719231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.719460 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:52.719632 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:52.719770 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 11:41:52.719951 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[5508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:52.730989 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:52.722997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.0.113.62:42691)
0806 11:41:52.723141 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:52.723145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:52.723158 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:41:52.723249 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:52.723251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:52.723288 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:52.723352 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:52.723884 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.724484 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.725205 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:52.725422 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:52.725539 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:52.727810 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:52.727813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:52.727815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:52.727839 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:52.729291 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.729719 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.729723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.729725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.729807 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:52.730101 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:52.730104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:52.730106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:52.730319 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:52.730467 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:52.730704 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 11:41:52.730823 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 11:41:52.731392 5580 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46132
I20260806 11:41:52.732031 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 5674
I20260806 11:41:52.742118 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 5852
I20260806 11:41:52.755259 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 6029
I20260806 11:41:52.767674 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 5533
2026-08-06T11:41:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2306 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[6228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[6228](info): set up 2 sockets
Aug 06 11:41:52 dist-test-slave-45dx krb5kdc[6228](info): commencing operation
krb5kdc: starting...
W20260806 11:41:53.977428 452 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:54 dist-test-slave-45dx krb5kdc[6228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:54.100401 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:42017
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:54.211719 6246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:54.211959 6246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:54.211995 6246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:54.215878 6246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:54.215930 6246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:54.215948 6246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:54.215965 6246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:54.215978 6246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:54.221202 6246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:42017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:54.222204 6246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:54.223194 6246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:54.231932 6255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:54.232008 6246 server_base.cc:1034] running on GCE node
W20260806 11:41:54.231930 6256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:54.231901 6260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:54.232798 6246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:54.233798 6246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:54.234980 6246 hybrid_clock.cc:648] HybridClock initialized: now 1786016514234955 us; error 37 us; skew 500 ppm
Aug 06 11:41:54 dist-test-slave-45dx krb5kdc[6228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:54.237859 6246 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:54.238565 6246 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:54.238657 6246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:54.238881 6246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:54.240928 6246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "19d27e3c14844247a753c30eb36b5757"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "ea617c100c2472c93436dfc7ad81daf7"
server_key_iv: "9286e20b2638d045c3cbbd345ec306e8"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.241468 6246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "19d27e3c14844247a753c30eb36b5757"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "ea617c100c2472c93436dfc7ad81daf7"
server_key_iv: "9286e20b2638d045c3cbbd345ec306e8"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.244964 6246 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:41:54.247224 6267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:54.248451 6246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:54.248592 6246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "19d27e3c14844247a753c30eb36b5757"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "ea617c100c2472c93436dfc7ad81daf7"
server_key_iv: "9286e20b2638d045c3cbbd345ec306e8"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.248700 6246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:54.267933 6246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:54.271693 6246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:54.271838 6246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:54.278632 6246 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:42017
I20260806 11:41:54.278890 6339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:42017 every 8 connection(s)
I20260806 11:41:54.279284 6246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:54.281956 6341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:54.287225 6341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757: Bootstrap starting.
I20260806 11:41:54.287717 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 6246
I20260806 11:41:54.287847 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:54.288106 452 external_mini_cluster.cc:1428] Setting key c04b563a260e58e31e1cf5ed87abf0dd
I20260806 11:41:54.289841 6341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:54.290573 6341 log.cc:824] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:54.292727 6341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757: No bootstrap required, opened a new log
Aug 06 11:41:54 dist-test-slave-45dx krb5kdc[6228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:54.297212 6341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d27e3c14844247a753c30eb36b5757" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42017 } }
I20260806 11:41:54.297487 6341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:54.297574 6341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d27e3c14844247a753c30eb36b5757, State: Initialized, Role: FOLLOWER
I20260806 11:41:54.298028 6341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d27e3c14844247a753c30eb36b5757" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42017 } }
I20260806 11:41:54.298202 6341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:54.298297 6341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:54.298439 6341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:54.299268 6341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d27e3c14844247a753c30eb36b5757" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42017 } }
I20260806 11:41:54.299593 6341 leader_election.cc:302] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d27e3c14844247a753c30eb36b5757; no voters:
I20260806 11:41:54.299894 6341 leader_election.cc:288] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:54.300011 6346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:54.300230 6346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [term 1 LEADER]: Becoming Leader. State: Replica: 19d27e3c14844247a753c30eb36b5757, State: Running, Role: LEADER
I20260806 11:41:54.300590 6346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d27e3c14844247a753c30eb36b5757" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42017 } }
I20260806 11:41:54.301201 6341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:54.302214 6349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d27e3c14844247a753c30eb36b5757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d27e3c14844247a753c30eb36b5757" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42017 } } }
I20260806 11:41:54.302337 6349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:54.302621 6350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d27e3c14844247a753c30eb36b5757. Latest consensus state: current_term: 1 leader_uuid: "19d27e3c14844247a753c30eb36b5757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d27e3c14844247a753c30eb36b5757" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42017 } } }
I20260806 11:41:54.302707 6350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:54.303084 6361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:54.303225 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:54.289473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.0.113.62:42017)
0806 11:41:54.289894 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:54.289912 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:54.289967 (+ 55us) server_negotiation.cc:407] Connection header received
0806 11:41:54.290597 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:54.290613 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:54.291008 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:54.291362 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:54.292236 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.293275 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.294464 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.294922 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.295040 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:54.297913 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:54.297918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:54.297930 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:54.297956 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:54.299836 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:54.300417 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:54.300423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:54.300429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:54.300566 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:54.300867 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:54.300871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:54.300873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:54.301348 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:54.301585 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:54.301884 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 11:41:54.302217 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":143,"threads_started":1}
I20260806 11:41:54.306756 6361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:54.311947 6361 catalog_manager.cc:1347] Generated new cluster ID: 3e2d0c3ebc4d45a7b1e648038a53b8f6
I20260806 11:41:54.312006 6361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:54.334430 6361 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:54.335389 6361 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:54.340970 6361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757: Generated new TSK 0
I20260806 11:41:54.341835 6361 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:41:54.377373 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42017
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:54.484814 6380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:54.485071 6380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:54.485148 6380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:54.489022 6380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:54.489097 6380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:54.489215 6380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:54.494014 6380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:54.494982 6380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:54.495939 6380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:54.505321 6386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:54.505297 6387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:54.505316 6389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:54.505699 6380 server_base.cc:1034] running on GCE node
I20260806 11:41:54.506114 6380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:54.506692 6380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:54.507876 6380 hybrid_clock.cc:648] HybridClock initialized: now 1786016514507863 us; error 25 us; skew 500 ppm
Aug 06 11:41:54 dist-test-slave-45dx krb5kdc[6228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:54.510592 6380 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:41:54.511482 6380 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:54.511575 6380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:54.511777 6380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:54.514402 6380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "3d34dc610476490f88744ab196f7945d"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "92faa05b9d3fb78ffc4088347e57550d"
server_key_iv: "316c0ef41422bcf1fb775f5811eb0ba5"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.515025 6380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "3d34dc610476490f88744ab196f7945d"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "92faa05b9d3fb78ffc4088347e57550d"
server_key_iv: "316c0ef41422bcf1fb775f5811eb0ba5"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.519776 6380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 11:41:54.522153 6396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:54.523339 6380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 11:41:54.523474 6380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "3d34dc610476490f88744ab196f7945d"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "92faa05b9d3fb78ffc4088347e57550d"
server_key_iv: "316c0ef41422bcf1fb775f5811eb0ba5"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.523577 6380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:54.563019 6380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:54.566437 6380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:54.566659 6380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:54.567297 6380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:54.568197 6380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:54.568248 6380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:54.568341 6380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:54.568377 6380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:54.577280 6380 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:43777
I20260806 11:41:54.577301 6565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:43777 every 8 connection(s)
I20260806 11:41:54.577900 6380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:54.578918 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 6380
I20260806 11:41:54.579048 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:54.579404 452 external_mini_cluster.cc:1428] Setting key b8d08a71b7159da5d66aa21e547d7f27
Aug 06 11:41:54 dist-test-slave-45dx krb5kdc[6228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:41:54.593155 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:54.579826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:34267 (local address 127.0.113.62:42017)
0806 11:41:54.580000 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:54.580005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:54.581024 (+ 1019us) server_negotiation.cc:407] Connection header received
0806 11:41:54.582515 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:54.582521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:54.582583 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:54.582678 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:54.583876 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.584474 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.586219 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.586457 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.586543 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:54.589572 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:54.589575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:54.589577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:54.589600 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:54.591089 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:54.591686 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:54.591689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:54.591691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:54.591770 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:54.592095 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:54.592098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:54.592099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:54.592391 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:54.592474 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:54.592927 (+ 453us) server_negotiation.cc:299] Negotiation successful
0806 11:41:54.593030 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:41:54.594146 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:54.580278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:42017 (local address 127.0.113.1:34267)
0806 11:41:54.580857 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:54.580925 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:54.582299 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:54.582858 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:54.582866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:54.583241 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:54.583723 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:54.583736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.584601 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.584608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:54.586088 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:54.586103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.586629 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.586633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:54.586887 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:54.587503 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:54.587546 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:54.589403 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:54.591221 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:54.591227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:54.591245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:54.591595 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:54.591856 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:54.591858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:54.591859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:54.592008 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:54.592488 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:54.592493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:54.592717 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 11:41:54.592873 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 11:41:54.593134 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":121,"threads_started":1}
I20260806 11:41:54.595429 6566 heartbeater.cc:347] Connected to a master server at 127.0.113.62:42017
I20260806 11:41:54.595681 6566 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:54.596252 6566 heartbeater.cc:511] Master 127.0.113.62:42017 requested a full tablet report, sending...
I20260806 11:41:54.597915 6290 ts_manager.cc:194] Registered new tserver with Master: 3d34dc610476490f88744ab196f7945d (127.0.113.1:43777)
I20260806 11:41:54.599110 6290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:34267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:41:54.611029 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42017
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:54.721859 6573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:54.722124 6573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:54.722201 6573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:54.726082 6573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:54.726154 6573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:54.726238 6573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:41:54.731588 6573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:54.732656 6573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:54.733629 6573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:54.742825 6582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:54.742816 6584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:54.742811 6581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:54.743089 6573 server_base.cc:1034] running on GCE node
I20260806 11:41:54.743561 6573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:54.744118 6573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:54.745278 6573 hybrid_clock.cc:648] HybridClock initialized: now 1786016514745267 us; error 59 us; skew 500 ppm
Aug 06 11:41:54 dist-test-slave-45dx krb5kdc[6228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:54.755257 6573 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:41:54.755970 6573 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:54.756085 6573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:54.756381 6573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:54.758473 6573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "18b0da13e3be4213826b57e63bba4994"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "40331a6a4afcaed20dfff867e40a5c4a"
server_key_iv: "c437f73f84267684cb63c03750b8acd1"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.759024 6573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "18b0da13e3be4213826b57e63bba4994"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "40331a6a4afcaed20dfff867e40a5c4a"
server_key_iv: "c437f73f84267684cb63c03750b8acd1"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.762609 6573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 11:41:54.764979 6595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:54.766201 6573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260806 11:41:54.766322 6573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "18b0da13e3be4213826b57e63bba4994"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "40331a6a4afcaed20dfff867e40a5c4a"
server_key_iv: "c437f73f84267684cb63c03750b8acd1"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.766433 6573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:54.781942 6573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:54.784687 6573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:54.784880 6573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:54.785420 6573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:54.786202 6573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:54.786249 6573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:54.786317 6573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:54.786353 6573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:54.795748 6573 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:34413
I20260806 11:41:54.795773 6755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:34413 every 8 connection(s)
I20260806 11:41:54.796291 6573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:41:54.797360 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 6573
I20260806 11:41:54.797500 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:41:54.797806 452 external_mini_cluster.cc:1428] Setting key 6a19304060d684f827d5d24dce207660
Aug 06 11:41:54 dist-test-slave-45dx krb5kdc[6228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:41:54.810753 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:54.797989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:56811 (local address 127.0.113.62:42017)
0806 11:41:54.798300 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:54.798307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:54.798945 (+ 638us) server_negotiation.cc:407] Connection header received
0806 11:41:54.800403 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:54.800406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:54.800465 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:54.800570 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:54.802008 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.802866 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.804000 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.804452 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.804585 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:54.807300 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:54.807303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:54.807305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:54.807328 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:54.808851 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:54.809321 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:54.809324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:54.809325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:54.809399 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:54.809758 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:54.809760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:54.809761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:54.809952 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:54.810029 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:54.810476 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 11:41:54.810628 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
I20260806 11:41:54.811502 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:54.798325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:42017 (local address 127.0.113.2:56811)
0806 11:41:54.798798 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:54.798847 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:54.800166 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:54.800735 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:54.800742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:54.801226 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:54.801842 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:54.801864 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.803011 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.803014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:54.803871 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:54.803882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:54.804605 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:54.804608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:54.804746 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:54.805340 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:54.805356 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:54.807132 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:54.808966 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:54.808971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:54.808980 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:54.809236 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:54.809482 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:54.809486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:54.809487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:54.809671 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:54.810043 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:54.810048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:54.810259 (+ 211us) client_negotiation.cc:769] Sending connection context
0806 11:41:54.810408 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 11:41:54.810633 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":147,"threads_started":1}
I20260806 11:41:54.812642 6756 heartbeater.cc:347] Connected to a master server at 127.0.113.62:42017
I20260806 11:41:54.812923 6756 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:54.813371 6756 heartbeater.cc:511] Master 127.0.113.62:42017 requested a full tablet report, sending...
I20260806 11:41:54.814441 6290 ts_manager.cc:194] Registered new tserver with Master: 18b0da13e3be4213826b57e63bba4994 (127.0.113.2:34413)
I20260806 11:41:54.814997 6290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:56811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:41:54.826769 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42017
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:54.936520 6766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:54.936772 6766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:54.936870 6766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:54.940824 6766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:54.940891 6766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:54.940970 6766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:41:54.946002 6766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:54.946877 6766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:54.947808 6766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:54.957067 6772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:54.957059 6773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:54.957059 6775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:54.957424 6766 server_base.cc:1034] running on GCE node
I20260806 11:41:54.957854 6766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:54.958465 6766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:54.959630 6766 hybrid_clock.cc:648] HybridClock initialized: now 1786016514959611 us; error 28 us; skew 500 ppm
Aug 06 11:41:54 dist-test-slave-45dx krb5kdc[6228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:54.962410 6766 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:41:54.963081 6766 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:54.963173 6766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:54.963398 6766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:54.965463 6766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "76dc07ae9e1148608138505c60cdf5a0"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "29038148ffabdb552f37d75f7cce16e5"
server_key_iv: "ce4283beb29bb5c5ff1d5d0a91b91ee6"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.966028 6766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "76dc07ae9e1148608138505c60cdf5a0"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "29038148ffabdb552f37d75f7cce16e5"
server_key_iv: "ce4283beb29bb5c5ff1d5d0a91b91ee6"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.969573 6766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:41:54.971828 6784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:54.973052 6766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:54.973191 6766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "76dc07ae9e1148608138505c60cdf5a0"
format_stamp: "Formatted at 2026-08-06 11:41:54 on dist-test-slave-45dx"
server_key: "29038148ffabdb552f37d75f7cce16e5"
server_key_iv: "ce4283beb29bb5c5ff1d5d0a91b91ee6"
server_key_version: "encryptionkey@0"
I20260806 11:41:54.973294 6766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:55.008229 6766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:55.011323 6766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:55.011516 6766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:55.012082 6766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:55.012974 6766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:55.013026 6766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.013067 6766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:55.013099 6766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.022482 6766 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:37751
I20260806 11:41:55.022508 6938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:37751 every 8 connection(s)
I20260806 11:41:55.023082 6766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
Aug 06 11:41:55 dist-test-slave-45dx krb5kdc[6228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:55.033123 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 6766
I20260806 11:41:55.033263 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:41:55.033689 452 external_mini_cluster.cc:1428] Setting key 0329ab62d581f17f051dfd7556e43ccf
I20260806 11:41:55.037575 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.024976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:50477 (local address 127.0.113.62:42017)
0806 11:41:55.025118 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.025122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.025894 (+ 772us) server_negotiation.cc:407] Connection header received
0806 11:41:55.027406 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.027410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.027473 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.027580 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:55.028889 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.029428 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.030585 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.030807 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.030921 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.033543 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:55.033547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:55.033551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:55.033581 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:55.035277 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:55.035904 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:55.035910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:55.035914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:55.036007 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:55.036422 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:55.036425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:55.036427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:55.036655 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:55.036751 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.037317 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.037447 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:41:55.038295 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.025274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:42017 (local address 127.0.113.3:50477)
0806 11:41:55.025732 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.025773 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:55.027176 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:55.027752 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:55.027758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:55.028193 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:55.028728 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.028742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.029562 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.029570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.030455 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.030464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.030962 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.030967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.031132 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.031668 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:55.031684 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:55.033325 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:55.035430 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:55.035436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:55.035449 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:55.035801 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:55.036118 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:55.036120 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:55.036122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:55.036259 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:55.036766 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:55.036771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:55.037002 (+ 231us) client_negotiation.cc:769] Sending connection context
0806 11:41:55.037152 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 11:41:55.037413 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":127,"threads_started":1}
I20260806 11:41:55.039425 6942 heartbeater.cc:347] Connected to a master server at 127.0.113.62:42017
I20260806 11:41:55.039652 6942 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:55.040097 6942 heartbeater.cc:511] Master 127.0.113.62:42017 requested a full tablet report, sending...
I20260806 11:41:55.041124 6290 ts_manager.cc:194] Registered new tserver with Master: 76dc07ae9e1148608138505c60cdf5a0 (127.0.113.3:37751)
I20260806 11:41:55.041695 6290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:50477
I20260806 11:41:55.047721 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:41:55.057096 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.049729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.0.113.62:42017)
0806 11:41:55.049844 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.049848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.050034 (+ 186us) server_negotiation.cc:407] Connection header received
0806 11:41:55.050156 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.050158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.050194 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.050261 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:55.050966 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.051435 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.052254 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.052527 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.052603 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.053951 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:55.053955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:55.053957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:55.053978 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:55.055453 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:55.055870 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:55.055873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:55.055875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:55.055956 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:55.056223 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:55.056226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:55.056227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:55.056518 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:55.056632 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.056875 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.056971 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 11:41:55.059801 6290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:41:55.062922 6290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:55.078573 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.071968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.0.113.1:43777)
0806 11:41:55.073116 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.073123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.073146 (+ 23us) server_negotiation.cc:407] Connection header received
0806 11:41:55.073250 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.073257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.073476 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.073656 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:55.074159 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.075264 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.076786 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.077735 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.077826 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.077962 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.078316 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.078415 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":144,"threads_started":1}
I20260806 11:41:55.078570 6958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.071742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:43777 (local address 127.0.0.1:48246)
0806 11:41:55.072526 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.072579 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:55.072713 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:55.073845 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:55.073852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:55.073875 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:55.074004 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.074009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.075406 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.075408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.076663 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.076673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.078028 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.078031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.078116 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.078157 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 11:41:55.078281 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 11:41:55.078405 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":78,"spinlock_wait_cycles":4224,"thread_start_us":80,"threads_started":1}
I20260806 11:41:55.078660 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.072568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.0.113.3:37751)
0806 11:41:55.073098 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.073110 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.073132 (+ 22us) server_negotiation.cc:407] Connection header received
0806 11:41:55.073228 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.073235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.073439 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.073598 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:55.074098 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.074921 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.076925 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.077690 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.077813 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.077963 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.078405 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.078509 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":134,"threads_started":1}
I20260806 11:41:55.078791 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.072014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:37751 (local address 127.0.0.1:40154)
0806 11:41:55.072737 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.072752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:55.072845 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:55.073787 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:55.073789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:55.073813 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:55.073958 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.073965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.075074 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.075081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.076788 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.076800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.077837 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.077843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.078053 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.078184 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 11:41:55.078265 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 11:41:55.078652 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":443,"thread_start_us":114,"threads_started":1}
I20260806 11:41:55.078588 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.071656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:34413 (local address 127.0.0.1:53364)
0806 11:41:55.072526 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.072579 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:55.072713 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:55.073612 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:55.073618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:55.073645 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:55.073795 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.073806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.075337 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.075340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.076788 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.076800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.077890 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.077894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.078104 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.078186 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 11:41:55.078262 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 11:41:55.078417 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":145,"spinlock_wait_cycles":6144,"thread_start_us":78,"threads_started":1}
I20260806 11:41:55.080569 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.072333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.0.113.2:34413)
0806 11:41:55.073179 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.073189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.073207 (+ 18us) server_negotiation.cc:407] Connection header received
0806 11:41:55.073272 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.073276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.073434 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.073598 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:55.073938 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.075182 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.076961 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.077730 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.080047 (+ 2317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.080203 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.080257 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.080402 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":124,"threads_started":1}
I20260806 11:41:55.082208 6470 tablet_service.cc:1467] Processing CreateTablet for tablet ae9638f124664b3699d1954ae7e294fc (DEFAULT_TABLE table=test-table [id=39532cc97862426dac401066d0321402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:55.082093 6851 tablet_service.cc:1467] Processing CreateTablet for tablet ae9638f124664b3699d1954ae7e294fc (DEFAULT_TABLE table=test-table [id=39532cc97862426dac401066d0321402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:55.083225 6851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9638f124664b3699d1954ae7e294fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:55.083220 6470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9638f124664b3699d1954ae7e294fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:55.083115 6666 tablet_service.cc:1467] Processing CreateTablet for tablet ae9638f124664b3699d1954ae7e294fc (DEFAULT_TABLE table=test-table [id=39532cc97862426dac401066d0321402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:55.084025 6666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9638f124664b3699d1954ae7e294fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:55.089672 6965 tablet_bootstrap.cc:492] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d: Bootstrap starting.
I20260806 11:41:55.091133 6967 tablet_bootstrap.cc:492] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: Bootstrap starting.
I20260806 11:41:55.091324 6968 tablet_bootstrap.cc:492] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: Bootstrap starting.
I20260806 11:41:55.091987 6965 tablet_bootstrap.cc:654] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:55.093314 6968 tablet_bootstrap.cc:654] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:55.093314 6967 tablet_bootstrap.cc:654] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:55.093721 6965 log.cc:824] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:55.094586 6967 log.cc:824] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:55.094586 6968 log.cc:824] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:55.095602 6965 tablet_bootstrap.cc:492] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d: No bootstrap required, opened a new log
I20260806 11:41:55.095803 6965 ts_tablet_manager.cc:1397] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 11:41:55.097733 6967 tablet_bootstrap.cc:492] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: No bootstrap required, opened a new log
I20260806 11:41:55.097944 6967 ts_tablet_manager.cc:1397] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 11:41:55.098344 6968 tablet_bootstrap.cc:492] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: No bootstrap required, opened a new log
I20260806 11:41:55.098534 6968 ts_tablet_manager.cc:1397] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 11:41:55.099659 6965 raft_consensus.cc:348] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } }
I20260806 11:41:55.099947 6965 raft_consensus.cc:374] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:55.100021 6965 raft_consensus.cc:729] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d34dc610476490f88744ab196f7945d, State: Initialized, Role: FOLLOWER
I20260806 11:41:55.100833 6965 consensus_queue.cc:260] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } }
I20260806 11:41:55.101167 6967 raft_consensus.cc:348] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } }
I20260806 11:41:55.101441 6967 raft_consensus.cc:374] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:55.101518 6967 raft_consensus.cc:729] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76dc07ae9e1148608138505c60cdf5a0, State: Initialized, Role: FOLLOWER
I20260806 11:41:55.101534 6566 heartbeater.cc:503] Master 127.0.113.62:42017 was elected leader, sending a full tablet report...
I20260806 11:41:55.102089 6967 consensus_queue.cc:260] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } }
I20260806 11:41:55.103060 6967 ts_tablet_manager.cc:1428] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 11:41:55.103145 6968 raft_consensus.cc:348] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } }
I20260806 11:41:55.103420 6968 raft_consensus.cc:374] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:55.103410 6942 heartbeater.cc:503] Master 127.0.113.62:42017 was elected leader, sending a full tablet report...
I20260806 11:41:55.103474 6968 raft_consensus.cc:729] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b0da13e3be4213826b57e63bba4994, State: Initialized, Role: FOLLOWER
I20260806 11:41:55.103896 6968 consensus_queue.cc:260] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } }
I20260806 11:41:55.104698 6965 ts_tablet_manager.cc:1428] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 11:41:55.104909 6968 ts_tablet_manager.cc:1428] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 11:41:55.105304 6756 heartbeater.cc:503] Master 127.0.113.62:42017 was elected leader, sending a full tablet report...
I20260806 11:41:55.110167 6973 raft_consensus.cc:482] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:55.110332 6973 raft_consensus.cc:504] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } }
I20260806 11:41:55.111251 6973 leader_election.cc:288] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18b0da13e3be4213826b57e63bba4994 (127.0.113.2:34413), 3d34dc610476490f88744ab196f7945d (127.0.113.1:43777)
I20260806 11:41:55.115115 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.111522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:34413 (local address 127.0.113.3:49227)
0806 11:41:55.111643 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.111660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:55.111747 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:55.112148 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:55.112151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:55.112171 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:55.112325 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.112333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.113189 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.113192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.113949 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.113958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.114693 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.114700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.114809 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.114837 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 11:41:55.114919 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 11:41:55.114973 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260806 11:41:55.115178 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.111612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:49227 (local address 127.0.113.2:34413)
0806 11:41:55.111911 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.111917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.111937 (+ 20us) server_negotiation.cc:407] Connection header received
0806 11:41:55.111992 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.111996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.112047 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.112126 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:55.112439 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.113067 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.114076 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.114568 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.114654 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.114703 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.114961 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.115049 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260806 11:41:55.115228 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.111741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:47849 (local address 127.0.113.1:43777)
0806 11:41:55.111857 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.111861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.112008 (+ 147us) server_negotiation.cc:407] Connection header received
0806 11:41:55.112149 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.112152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.112195 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.112275 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:41:55.112623 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.113089 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.114255 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.114662 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.114761 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.114816 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.115048 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.115114 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:41:55.115448 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.111522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:43777 (local address 127.0.113.3:47849)
0806 11:41:55.111909 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.111927 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:55.112042 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:55.112297 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:55.112352 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:55.112372 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:41:55.112516 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.112523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.113229 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.113231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.114122 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:55.114136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.114791 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.114794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:55.114889 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.114938 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 11:41:55.115019 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 11:41:55.115083 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":39,"thread_start_us":136,"threads_started":1}
I20260806 11:41:55.115761 6696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9638f124664b3699d1954ae7e294fc" candidate_uuid: "76dc07ae9e1148608138505c60cdf5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0da13e3be4213826b57e63bba4994" is_pre_election: true
I20260806 11:41:55.116014 6696 raft_consensus.cc:2454] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76dc07ae9e1148608138505c60cdf5a0 in term 0.
I20260806 11:41:55.116039 6499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9638f124664b3699d1954ae7e294fc" candidate_uuid: "76dc07ae9e1148608138505c60cdf5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d34dc610476490f88744ab196f7945d" is_pre_election: true
I20260806 11:41:55.116356 6499 raft_consensus.cc:2454] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76dc07ae9e1148608138505c60cdf5a0 in term 0.
I20260806 11:41:55.116552 6788 leader_election.cc:302] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b0da13e3be4213826b57e63bba4994, 76dc07ae9e1148608138505c60cdf5a0; no voters:
I20260806 11:41:55.116806 6973 raft_consensus.cc:2781] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:55.116892 6973 raft_consensus.cc:482] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:55.116937 6973 raft_consensus.cc:3037] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:55.117919 6973 raft_consensus.cc:504] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } }
I20260806 11:41:55.118256 6973 leader_election.cc:288] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [CANDIDATE]: Term 1 election: Requested vote from peers 18b0da13e3be4213826b57e63bba4994 (127.0.113.2:34413), 3d34dc610476490f88744ab196f7945d (127.0.113.1:43777)
I20260806 11:41:55.118600 6696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9638f124664b3699d1954ae7e294fc" candidate_uuid: "76dc07ae9e1148608138505c60cdf5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0da13e3be4213826b57e63bba4994"
I20260806 11:41:55.118660 6499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9638f124664b3699d1954ae7e294fc" candidate_uuid: "76dc07ae9e1148608138505c60cdf5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d34dc610476490f88744ab196f7945d"
I20260806 11:41:55.118723 6696 raft_consensus.cc:3037] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:55.118757 6499 raft_consensus.cc:3037] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:55.119994 6499 raft_consensus.cc:2454] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76dc07ae9e1148608138505c60cdf5a0 in term 1.
I20260806 11:41:55.119994 6696 raft_consensus.cc:2454] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76dc07ae9e1148608138505c60cdf5a0 in term 1.
I20260806 11:41:55.120332 6788 leader_election.cc:302] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b0da13e3be4213826b57e63bba4994, 76dc07ae9e1148608138505c60cdf5a0; no voters:
I20260806 11:41:55.120589 6973 raft_consensus.cc:2781] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:55.120810 6973 raft_consensus.cc:686] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 1 LEADER]: Becoming Leader. State: Replica: 76dc07ae9e1148608138505c60cdf5a0, State: Running, Role: LEADER
I20260806 11:41:55.121121 6973 consensus_queue.cc:237] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } }
I20260806 11:41:55.124089 6288 catalog_manager.cc:5495] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76dc07ae9e1148608138505c60cdf5a0 (127.0.113.3). New cstate: current_term: 1 leader_uuid: "76dc07ae9e1148608138505c60cdf5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76dc07ae9e1148608138505c60cdf5a0" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 37751 } health_report { overall_health: HEALTHY } } }
I20260806 11:41:55.140938 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.136137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.0.113.3:37751)
0806 11:41:55.136274 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.136280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.136330 (+ 50us) server_negotiation.cc:407] Connection header received
0806 11:41:55.136524 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.136527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.136592 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.136740 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:41:55.137172 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.137875 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.139313 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:55.139653 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:55.139789 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:55.140115 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:41:55.140549 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:41:55.140674 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.140736 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.140788 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 11:41:55.146226 6696 raft_consensus.cc:1261] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [term 1 FOLLOWER]: Refusing update from remote peer 76dc07ae9e1148608138505c60cdf5a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:55.146226 6499 raft_consensus.cc:1261] T ae9638f124664b3699d1954ae7e294fc P 3d34dc610476490f88744ab196f7945d [term 1 FOLLOWER]: Refusing update from remote peer 76dc07ae9e1148608138505c60cdf5a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:55.146920 6976 consensus_queue.cc:1035] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d34dc610476490f88744ab196f7945d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:55.147099 6973 consensus_queue.cc:1035] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b0da13e3be4213826b57e63bba4994" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:55.154842 6985 mvcc.cc:205] Tried to move back new op lower bound from 7315523646032551936 to 7315523645939077120. Current Snapshot: MvccSnapshot[applied={T|T < 7315523646032551936}]
I20260806 11:41:55.155303 6984 mvcc.cc:205] Tried to move back new op lower bound from 7315523646032551936 to 7315523645939077120. Current Snapshot: MvccSnapshot[applied={T|T < 7315523646032551936}]
I20260806 11:41:55.156258 6986 mvcc.cc:205] Tried to move back new op lower bound from 7315523646032551936 to 7315523645939077120. Current Snapshot: MvccSnapshot[applied={T|T < 7315523646032551936}]
I20260806 11:41:55.162796 6288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:55.162938 6288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:55.165244 6288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19d27e3c14844247a753c30eb36b5757: Sending DeleteTablet for 3 replicas of tablet ae9638f124664b3699d1954ae7e294fc
I20260806 11:41:55.166023 6666 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9638f124664b3699d1954ae7e294fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:55 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:53364
I20260806 11:41:55.166215 6470 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9638f124664b3699d1954ae7e294fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:55 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:48246
I20260806 11:41:55.166584 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 6380
I20260806 11:41:55.166677 6851 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9638f124664b3699d1954ae7e294fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:55 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:40154
I20260806 11:41:55.174288 6995 tablet_replica.cc:330] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: stopping tablet replica
I20260806 11:41:55.174692 6995 raft_consensus.cc:2229] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:55.175038 6995 raft_consensus.cc:2258] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:55.175413 6998 tablet_replica.cc:330] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: stopping tablet replica
I20260806 11:41:55.175681 6998 raft_consensus.cc:2229] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:41:55.176101 6998 raft_consensus.cc:2258] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:55.177132 6995 ts_tablet_manager.cc:1905] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:55.177526 6998 ts_tablet_manager.cc:1905] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:55.180166 6998 ts_tablet_manager.cc:1918] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:41:55.180266 6998 log.cc:1196] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/ae9638f124664b3699d1954ae7e294fc
I20260806 11:41:55.180635 6998 ts_tablet_manager.cc:1939] T ae9638f124664b3699d1954ae7e294fc P 76dc07ae9e1148608138505c60cdf5a0: Deleting consensus metadata
I20260806 11:41:55.181061 6995 ts_tablet_manager.cc:1918] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:41:55.181233 6995 log.cc:1196] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/ae9638f124664b3699d1954ae7e294fc
W20260806 11:41:55.181267 6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:43777 (error 108)
I20260806 11:41:55.181685 6995 ts_tablet_manager.cc:1939] T ae9638f124664b3699d1954ae7e294fc P 18b0da13e3be4213826b57e63bba4994: Deleting consensus metadata
I20260806 11:41:55.181803 6271 catalog_manager.cc:4841] TS 76dc07ae9e1148608138505c60cdf5a0 (127.0.113.3:37751): tablet ae9638f124664b3699d1954ae7e294fc (table test-table [id=39532cc97862426dac401066d0321402]) successfully deleted
I20260806 11:41:55.182291 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 6573
I20260806 11:41:55.196098 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.195728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:43777 (local address 127.0.0.1:48256)
0806 11:41:55.195927 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.196001 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:43777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260806 11:41:55.196269 6270 catalog_manager.cc:4584] TS 3d34dc610476490f88744ab196f7945d (127.0.113.1:43777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae9638f124664b3699d1954ae7e294fc: Network error: Client connection negotiation failed: client connection to 127.0.113.1:43777: connect: Connection refused (error 111)
I20260806 11:41:55.205338 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 6766
I20260806 11:41:55.208936 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.205521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:34413 (local address 127.0.0.1:53378)
0806 11:41:55.208742 (+ 3221us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.208822 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.2:34413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3108}
I20260806 11:41:55.228041 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 6246
2026-08-06T11:41:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2456 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T11:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:55Z Disabled control of system clock
I20260806 11:41:55.259858 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:45033
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:45033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 11:41:55.367203 7005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:55.367468 7005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:55.367503 7005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:55.371241 7005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:55.371294 7005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:55.371315 7005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:55.371330 7005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:55.371344 7005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:55.375887 7005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:45033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:45033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:55.376834 7005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:55.378083 7005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:55.386263 7013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:55.386451 7005 server_base.cc:1034] running on GCE node
W20260806 11:41:55.386272 7012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:55.386272 7016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:55.387171 7005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:55.388182 7005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:55.389400 7005 hybrid_clock.cc:648] HybridClock initialized: now 1786016515389375 us; error 42 us; skew 500 ppm
I20260806 11:41:55.391256 7005 webserver.cc:460] Webserver started at http://127.0.113.62:43533/ using document root <none> and password file <none>
I20260806 11:41:55.391819 7005 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:55.391916 7005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:55.392148 7005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:55.394153 7005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "9703d78042f9499dba1fe6cde9299f0e"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "ca8762dded74c1b158adcbb37753ee94"
server_key_iv: "7b90929d289c94a57f4c6b45e7fb2b18"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.394789 7005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "9703d78042f9499dba1fe6cde9299f0e"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "ca8762dded74c1b158adcbb37753ee94"
server_key_iv: "7b90929d289c94a57f4c6b45e7fb2b18"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.398381 7005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:41:55.400846 7026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.402127 7005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:55.402290 7005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "9703d78042f9499dba1fe6cde9299f0e"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "ca8762dded74c1b158adcbb37753ee94"
server_key_iv: "7b90929d289c94a57f4c6b45e7fb2b18"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.402410 7005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:55.407040 7005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:55.407637 7005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:55.407835 7005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:55.414789 7108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:45033 every 8 connection(s)
I20260806 11:41:55.414796 7005 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:45033
I20260806 11:41:55.415733 7005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:55.418975 7110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:55.424266 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e: Bootstrap starting.
I20260806 11:41:55.425318 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 7005
I20260806 11:41:55.425468 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:55.425771 452 external_mini_cluster.cc:1428] Setting key e0ad48f7c75eeb9b7287e1995d79c4be
I20260806 11:41:55.427138 7110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:55.427884 7110 log.cc:824] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:55.430060 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e: No bootstrap required, opened a new log
I20260806 11:41:55.431277 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.426947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.0.113.62:45033)
0806 11:41:55.427611 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.427627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.427663 (+ 36us) server_negotiation.cc:407] Connection header received
0806 11:41:55.428439 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.428462 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.428815 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.429192 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:55.430012 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:55.430017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:55.430029 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:55.430040 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:55.430096 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:55.430134 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:55.430248 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.430359 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.430605 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":141,"threads_started":1}
I20260806 11:41:55.433738 7110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d78042f9499dba1fe6cde9299f0e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 45033 } }
I20260806 11:41:55.433943 7110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:55.434016 7110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9703d78042f9499dba1fe6cde9299f0e, State: Initialized, Role: FOLLOWER
I20260806 11:41:55.434450 7110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d78042f9499dba1fe6cde9299f0e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 45033 } }
I20260806 11:41:55.434607 7110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:55.434695 7110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:55.434780 7110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:55.435750 7110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d78042f9499dba1fe6cde9299f0e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 45033 } }
I20260806 11:41:55.436081 7110 leader_election.cc:302] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9703d78042f9499dba1fe6cde9299f0e; no voters:
I20260806 11:41:55.436443 7110 leader_election.cc:288] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:55.436537 7119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:55.436769 7119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [term 1 LEADER]: Becoming Leader. State: Replica: 9703d78042f9499dba1fe6cde9299f0e, State: Running, Role: LEADER
I20260806 11:41:55.437088 7119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d78042f9499dba1fe6cde9299f0e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 45033 } }
I20260806 11:41:55.437654 7110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:55.438961 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9703d78042f9499dba1fe6cde9299f0e. Latest consensus state: current_term: 1 leader_uuid: "9703d78042f9499dba1fe6cde9299f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d78042f9499dba1fe6cde9299f0e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 45033 } } }
I20260806 11:41:55.438961 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9703d78042f9499dba1fe6cde9299f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d78042f9499dba1fe6cde9299f0e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 45033 } } }
I20260806 11:41:55.439201 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:55.439227 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:55.439998 7136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:55.442606 7136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:55.447554 7136 catalog_manager.cc:1347] Generated new cluster ID: df94263090294507ae5aee005777f0a8
I20260806 11:41:55.447618 7136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:55.458160 7136 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:55.458498 7136 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:55.465168 7136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e: Generated new TSK 0
I20260806 11:41:55.465855 7136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:41:55.468590 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:45033
--builtin_ntp_servers=127.0.113.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 11:41:55.590008 7144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:55.590299 7144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:55.590370 7144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:55.594687 7144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:55.594775 7144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:55.594877 7144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:55.600636 7144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:45033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:55.601814 7144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:55.603235 7144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:55.613818 7153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:55.613863 7144 server_base.cc:1034] running on GCE node
W20260806 11:41:55.613813 7152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:55.613813 7155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:55.614423 7144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:55.615032 7144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:55.616204 7144 hybrid_clock.cc:648] HybridClock initialized: now 1786016515616165 us; error 53 us; skew 500 ppm
I20260806 11:41:55.618181 7144 webserver.cc:460] Webserver started at http://127.0.113.1:34909/ using document root <none> and password file <none>
I20260806 11:41:55.618768 7144 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:55.618834 7144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:55.619048 7144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:55.621246 7144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "ca3adfbbb6c0417aa331271e90a27b4c"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "0395f8521b20b96121366f036a0fd57a"
server_key_iv: "0e556cdf4cbe52a2a1f341424e5e8b1b"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.622006 7144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "ca3adfbbb6c0417aa331271e90a27b4c"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "0395f8521b20b96121366f036a0fd57a"
server_key_iv: "0e556cdf4cbe52a2a1f341424e5e8b1b"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.625748 7144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:41:55.628240 7165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.629490 7144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:55.629629 7144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "ca3adfbbb6c0417aa331271e90a27b4c"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "0395f8521b20b96121366f036a0fd57a"
server_key_iv: "0e556cdf4cbe52a2a1f341424e5e8b1b"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.629745 7144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:55.634374 7144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:55.634985 7144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:55.635181 7144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:55.635743 7144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:55.636739 7144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:55.636811 7144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.636881 7144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:55.636935 7144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.646235 7144 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:44419
I20260806 11:41:55.646262 7333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:44419 every 8 connection(s)
I20260806 11:41:55.647178 7144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:55.653730 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.649136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:33767 (local address 127.0.113.62:45033)
0806 11:41:55.649285 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.649289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.650321 (+ 1032us) server_negotiation.cc:407] Connection header received
0806 11:41:55.651893 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.651896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.651938 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.652015 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:55.653222 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:55.653225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:55.653227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:55.653240 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:55.653260 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:55.653267 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:55.653347 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.653582 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.653621 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:41:55.654649 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.649466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:45033 (local address 127.0.113.1:33767)
0806 11:41:55.650138 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.650178 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:55.651693 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:55.652205 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:55.652211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:55.652297 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:55.652964 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:41:55.652965 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:55.653014 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:41:55.653080 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:55.653409 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:55.653421 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:55.653434 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 11:41:55.653620 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 11:41:55.653848 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260806 11:41:55.655895 7335 heartbeater.cc:347] Connected to a master server at 127.0.113.62:45033
I20260806 11:41:55.656128 7335 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:55.656622 7335 heartbeater.cc:511] Master 127.0.113.62:45033 requested a full tablet report, sending...
I20260806 11:41:55.656760 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 7144
I20260806 11:41:55.656912 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:55.657266 452 external_mini_cluster.cc:1428] Setting key 29bfd278310a934b0b1c45294025ff50
I20260806 11:41:55.658303 7058 ts_manager.cc:194] Registered new tserver with Master: ca3adfbbb6c0417aa331271e90a27b4c (127.0.113.1:44419)
I20260806 11:41:55.659942 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:45033
--builtin_ntp_servers=127.0.113.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 11:41:55.773061 7342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:55.773352 7342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:55.773442 7342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:55.777424 7342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:55.777506 7342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:55.777597 7342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:41:55.782267 7342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:45033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:55.783313 7342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:55.784703 7342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:55.794205 7349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:55.794230 7352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:55.794239 7350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:55.794549 7342 server_base.cc:1034] running on GCE node
I20260806 11:41:55.794970 7342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:55.795537 7342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:55.796698 7342 hybrid_clock.cc:648] HybridClock initialized: now 1786016515796676 us; error 32 us; skew 500 ppm
I20260806 11:41:55.798586 7342 webserver.cc:460] Webserver started at http://127.0.113.2:35649/ using document root <none> and password file <none>
I20260806 11:41:55.799180 7342 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:55.799243 7342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:55.799501 7342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:55.801580 7342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "0342805529ae4c0aa7a5f1ada25c2449"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "8395968e55cd66759e2fb637155caa32"
server_key_iv: "a0fc60e2bd5f1fedf1c989cfccf4737e"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.802179 7342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "0342805529ae4c0aa7a5f1ada25c2449"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "8395968e55cd66759e2fb637155caa32"
server_key_iv: "a0fc60e2bd5f1fedf1c989cfccf4737e"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.805817 7342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:41:55.808599 7362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.809841 7342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:41:55.809996 7342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "0342805529ae4c0aa7a5f1ada25c2449"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "8395968e55cd66759e2fb637155caa32"
server_key_iv: "a0fc60e2bd5f1fedf1c989cfccf4737e"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.810113 7342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:55.814775 7342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:55.815382 7342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:55.815565 7342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:55.816098 7342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:55.817075 7342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:55.817144 7342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.817219 7342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:55.817269 7342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.826027 7342 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:36693
I20260806 11:41:55.826035 7527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:36693 every 8 connection(s)
I20260806 11:41:55.826973 7342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:41:55.832962 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.828774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:52849 (local address 127.0.113.62:45033)
0806 11:41:55.828945 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:55.828950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:55.830007 (+ 1057us) server_negotiation.cc:407] Connection header received
0806 11:41:55.831262 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:55.831264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:55.831300 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:55.831375 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:55.832491 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:55.832493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:55.832496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:55.832507 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:55.832526 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:55.832553 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:55.832650 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:55.832813 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 11:41:55.832849 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 11:41:55.833840 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:55.829181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:45033 (local address 127.0.113.2:52849)
0806 11:41:55.829859 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:55.829904 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:55.831083 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:55.831563 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:55.831570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:55.831651 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:55.832161 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:41:55.832163 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:55.832201 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:41:55.832265 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:55.832657 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:55.832668 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:55.832678 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 11:41:55.832823 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 11:41:55.833039 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260806 11:41:55.834918 7528 heartbeater.cc:347] Connected to a master server at 127.0.113.62:45033
I20260806 11:41:55.835152 7528 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:55.835551 7528 heartbeater.cc:511] Master 127.0.113.62:45033 requested a full tablet report, sending...
I20260806 11:41:55.836611 7058 ts_manager.cc:194] Registered new tserver with Master: 0342805529ae4c0aa7a5f1ada25c2449 (127.0.113.2:36693)
I20260806 11:41:55.836865 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 7342
I20260806 11:41:55.837014 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:41:55.837292 452 external_mini_cluster.cc:1428] Setting key a9bfbca47fe74c5fb4059c1d3f768018
I20260806 11:41:55.839165 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:45033
--builtin_ntp_servers=127.0.113.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 11:41:55.954453 7532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:55.954730 7532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:55.954839 7532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:55.958860 7532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:55.958945 7532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:55.959029 7532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:41:55.963764 7532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:45033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:55.965008 7532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:55.966475 7532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:55.976936 7543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:55.976941 7541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:55.976961 7540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:55.977160 7532 server_base.cc:1034] running on GCE node
I20260806 11:41:55.977581 7532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:55.978246 7532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:55.979414 7532 hybrid_clock.cc:648] HybridClock initialized: now 1786016515979391 us; error 34 us; skew 500 ppm
I20260806 11:41:55.981258 7532 webserver.cc:460] Webserver started at http://127.0.113.3:40853/ using document root <none> and password file <none>
I20260806 11:41:55.981877 7532 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:55.981942 7532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:55.982194 7532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:55.984462 7532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "b1deace4b7814a7ebe88c7d4d8c61454"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "72c4458ccfefdef736cddff9316adbbe"
server_key_iv: "93f06784da7bbb061c0728f01638e24a"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.985030 7532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "b1deace4b7814a7ebe88c7d4d8c61454"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "72c4458ccfefdef736cddff9316adbbe"
server_key_iv: "93f06784da7bbb061c0728f01638e24a"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.989005 7532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 11:41:55.991541 7553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.992841 7532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:55.992973 7532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "b1deace4b7814a7ebe88c7d4d8c61454"
format_stamp: "Formatted at 2026-08-06 11:41:55 on dist-test-slave-45dx"
server_key: "72c4458ccfefdef736cddff9316adbbe"
server_key_iv: "93f06784da7bbb061c0728f01638e24a"
server_key_version: "encryptionkey@0"
I20260806 11:41:55.993099 7532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:55.997618 7532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:55.998198 7532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:55.998386 7532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:55.998950 7532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:55.999838 7532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:55.999913 7532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:55.999981 7532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:56.000034 7532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:56.010290 7532 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:39351
I20260806 11:41:56.010298 7722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:39351 every 8 connection(s)
I20260806 11:41:56.011204 7532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:41:56.015903 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 7532
I20260806 11:41:56.016044 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:41:56.016472 452 external_mini_cluster.cc:1428] Setting key 58ee6fa6e5c5f4dd1ce7f5d31b40f194
I20260806 11:41:56.018085 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.013145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:55221 (local address 127.0.113.62:45033)
0806 11:41:56.013299 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.013303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.014568 (+ 1265us) server_negotiation.cc:407] Connection header received
0806 11:41:56.016002 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.016008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.016056 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.016131 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.017548 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.017551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.017554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.017567 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.017593 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.017606 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.017679 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.017872 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.017942 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 11:41:56.019068 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.013609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:45033 (local address 127.0.113.3:55221)
0806 11:41:56.014420 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:56.014462 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:56.015765 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:56.016363 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:56.016375 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:56.016492 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:56.017216 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:41:56.017219 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:56.017272 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:41:56.017362 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:56.017700 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:56.017709 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:56.017716 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 11:41:56.017919 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 11:41:56.018176 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":146,"threads_started":1}
I20260806 11:41:56.020133 7723 heartbeater.cc:347] Connected to a master server at 127.0.113.62:45033
I20260806 11:41:56.020419 7723 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:56.020823 7723 heartbeater.cc:511] Master 127.0.113.62:45033 requested a full tablet report, sending...
I20260806 11:41:56.021808 7058 ts_manager.cc:194] Registered new tserver with Master: b1deace4b7814a7ebe88c7d4d8c61454 (127.0.113.3:39351)
I20260806 11:41:56.030359 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:41:56.033887 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.032239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.0.113.62:45033)
0806 11:41:56.032421 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.032424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.032543 (+ 119us) server_negotiation.cc:407] Connection header received
0806 11:41:56.032628 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.032630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.032662 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.032725 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.033511 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.033514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.033516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.033525 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.033542 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.033549 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.033615 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.033702 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.033735 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 11:41:56.036629 7058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:41:56.039915 7058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:41:56.051573 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.048264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:44419 (local address 127.0.0.1:36812)
0806 11:41:56.049354 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:56.049421 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:56.049518 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:56.050070 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:56.050079 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:56.050154 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:56.050784 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:41:56.050788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:56.050808 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:41:56.050873 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:56.051219 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:56.051229 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:56.051234 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 11:41:56.051362 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 11:41:56.051431 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":125,"threads_started":1}
I20260806 11:41:56.051889 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.048990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.0.113.1:44419)
0806 11:41:56.049441 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.049454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.049486 (+ 32us) server_negotiation.cc:407] Connection header received
0806 11:41:56.049674 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.049681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.049899 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.050038 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.051021 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.051031 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.051045 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.051065 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.051100 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.051124 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.051246 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.051401 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.051520 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":146,"threads_started":1}
I20260806 11:41:56.052161 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.048511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.0.113.3:39351)
0806 11:41:56.049089 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.049102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.049637 (+ 535us) server_negotiation.cc:407] Connection header received
0806 11:41:56.049829 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.049836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.050018 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.050149 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.051502 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.051508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.051519 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.051539 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.051577 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.051625 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.051731 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.051908 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.051979 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":118,"threads_started":1}
I20260806 11:41:56.052484 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.048657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:36693 (local address 127.0.0.1:47980)
0806 11:41:56.049725 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:56.049737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:56.049807 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:56.050834 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:56.050836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:56.050849 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:56.051776 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:41:56.051778 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:56.051789 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:41:56.051819 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:56.052233 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:56.052236 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:56.052239 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 11:41:56.052332 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 11:41:56.052368 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":106,"threads_started":1}
I20260806 11:41:56.052733 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.048384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:39351 (local address 127.0.0.1:56550)
0806 11:41:56.049527 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:56.049543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:56.049697 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:56.050827 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:56.050830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:56.050844 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:56.051289 (+ 445us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:41:56.051292 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:56.051303 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:41:56.051336 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:56.051782 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:56.051787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:56.051792 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 11:41:56.052404 (+ 612us) client_negotiation.cc:240] Negotiation successful
0806 11:41:56.052441 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":98,"threads_started":1}
I20260806 11:41:56.052899 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.049087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.0.113.2:36693)
0806 11:41:56.049698 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.049711 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.049911 (+ 200us) server_negotiation.cc:407] Connection header received
0806 11:41:56.049975 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.049983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.050145 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.050263 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.051975 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.051981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.051995 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.052011 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.052054 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.052086 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.052187 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.052654 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.052721 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":116,"threads_started":1}
I20260806 11:41:56.055161 7234 tablet_service.cc:1467] Processing CreateTablet for tablet f703801934844cb092911d766741b70a (DEFAULT_TABLE table=test-table [id=7bc096d5a95e47ed85e6ff9c98d97db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:56.055388 7627 tablet_service.cc:1467] Processing CreateTablet for tablet f703801934844cb092911d766741b70a (DEFAULT_TABLE table=test-table [id=7bc096d5a95e47ed85e6ff9c98d97db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:56.055624 7424 tablet_service.cc:1467] Processing CreateTablet for tablet f703801934844cb092911d766741b70a (DEFAULT_TABLE table=test-table [id=7bc096d5a95e47ed85e6ff9c98d97db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:41:56.056281 7234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f703801934844cb092911d766741b70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:56.056466 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f703801934844cb092911d766741b70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:56.056475 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f703801934844cb092911d766741b70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:56.062410 7754 tablet_bootstrap.cc:492] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c: Bootstrap starting.
I20260806 11:41:56.063844 7755 tablet_bootstrap.cc:492] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449: Bootstrap starting.
I20260806 11:41:56.063844 7756 tablet_bootstrap.cc:492] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: Bootstrap starting.
I20260806 11:41:56.064981 7754 tablet_bootstrap.cc:654] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:56.065814 7754 log.cc:824] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:56.065985 7755 tablet_bootstrap.cc:654] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:56.065985 7756 tablet_bootstrap.cc:654] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:56.066833 7755 log.cc:824] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:56.066833 7756 log.cc:824] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:56.067909 7754 tablet_bootstrap.cc:492] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c: No bootstrap required, opened a new log
I20260806 11:41:56.068112 7754 ts_tablet_manager.cc:1397] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 11:41:56.068768 7755 tablet_bootstrap.cc:492] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449: No bootstrap required, opened a new log
I20260806 11:41:56.068946 7755 ts_tablet_manager.cc:1397] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260806 11:41:56.069232 7756 tablet_bootstrap.cc:492] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: No bootstrap required, opened a new log
I20260806 11:41:56.069404 7756 ts_tablet_manager.cc:1397] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 11:41:56.072412 7754 raft_consensus.cc:348] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } }
I20260806 11:41:56.072645 7754 raft_consensus.cc:374] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:56.072697 7754 raft_consensus.cc:729] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3adfbbb6c0417aa331271e90a27b4c, State: Initialized, Role: FOLLOWER
I20260806 11:41:56.073163 7754 consensus_queue.cc:260] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } }
I20260806 11:41:56.073379 7755 raft_consensus.cc:348] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } }
I20260806 11:41:56.073645 7755 raft_consensus.cc:374] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:56.073527 7756 raft_consensus.cc:348] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } }
I20260806 11:41:56.073745 7756 raft_consensus.cc:374] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:56.073802 7756 raft_consensus.cc:729] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1deace4b7814a7ebe88c7d4d8c61454, State: Initialized, Role: FOLLOWER
I20260806 11:41:56.073935 7335 heartbeater.cc:503] Master 127.0.113.62:45033 was elected leader, sending a full tablet report...
I20260806 11:41:56.074165 7754 ts_tablet_manager.cc:1428] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 11:41:56.074262 7756 consensus_queue.cc:260] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } }
I20260806 11:41:56.073736 7755 raft_consensus.cc:729] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0342805529ae4c0aa7a5f1ada25c2449, State: Initialized, Role: FOLLOWER
I20260806 11:41:56.074893 7723 heartbeater.cc:503] Master 127.0.113.62:45033 was elected leader, sending a full tablet report...
I20260806 11:41:56.074812 7755 consensus_queue.cc:260] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } }
I20260806 11:41:56.075222 7756 ts_tablet_manager.cc:1428] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 11:41:56.075712 7755 ts_tablet_manager.cc:1428] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 11:41:56.076114 7528 heartbeater.cc:503] Master 127.0.113.62:45033 was elected leader, sending a full tablet report...
W20260806 11:41:56.077950 7530 tablet.cc:2369] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:41:56.148406 7338 tablet.cc:2369] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:41:56.170542 7762 raft_consensus.cc:482] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:41:56.170783 7762 raft_consensus.cc:504] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } }
I20260806 11:41:56.172026 7762 leader_election.cc:288] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1deace4b7814a7ebe88c7d4d8c61454 (127.0.113.3:39351), 0342805529ae4c0aa7a5f1ada25c2449 (127.0.113.2:36693)
I20260806 11:41:56.174330 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.172415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:49043 (local address 127.0.113.3:39351)
0806 11:41:56.172573 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.172578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.172846 (+ 268us) server_negotiation.cc:407] Connection header received
0806 11:41:56.172952 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.172956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.172997 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.173083 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.173838 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.173842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.173845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.173858 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.173896 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.173910 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.174012 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.174116 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.174176 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 11:41:56.174332 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.172292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:39351 (local address 127.0.113.1:49043)
0806 11:41:56.172725 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:56.172738 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:56.172855 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:56.173102 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:56.173104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:56.173119 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:56.173667 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:41:56.173670 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:56.173684 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:41:56.173720 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:56.174019 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:56.174022 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:56.174026 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 11:41:56.174122 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 11:41:56.174165 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":106,"threads_started":1}
I20260806 11:41:56.174448 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.172531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:50831 (local address 127.0.113.2:36693)
0806 11:41:56.172666 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.172671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.172916 (+ 245us) server_negotiation.cc:407] Connection header received
0806 11:41:56.173111 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.173115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.173154 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.173206 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.174035 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.174039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.174042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.174057 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.174082 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.174096 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.174182 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.174274 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.174311 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 11:41:56.174540 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.172292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:36693 (local address 127.0.113.1:50831)
0806 11:41:56.172789 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:56.172801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:56.172993 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:56.173323 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:56.173326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:56.173340 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:56.173838 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:41:56.173840 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:56.173851 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:41:56.173879 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:56.174188 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:56.174191 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:56.174194 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 11:41:56.174280 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 11:41:56.174313 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":50,"thread_start_us":83,"threads_started":1}
I20260806 11:41:56.175216 7454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f703801934844cb092911d766741b70a" candidate_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0342805529ae4c0aa7a5f1ada25c2449" is_pre_election: true
I20260806 11:41:56.175297 7657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f703801934844cb092911d766741b70a" candidate_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" is_pre_election: true
I20260806 11:41:56.175554 7454 raft_consensus.cc:2454] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3adfbbb6c0417aa331271e90a27b4c in term 0.
I20260806 11:41:56.175608 7657 raft_consensus.cc:2454] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3adfbbb6c0417aa331271e90a27b4c in term 0.
I20260806 11:41:56.176074 7170 leader_election.cc:302] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0342805529ae4c0aa7a5f1ada25c2449, ca3adfbbb6c0417aa331271e90a27b4c; no voters:
I20260806 11:41:56.176400 7762 raft_consensus.cc:2781] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:41:56.176506 7762 raft_consensus.cc:482] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:41:56.176555 7762 raft_consensus.cc:3037] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:56.208575 7762 raft_consensus.cc:504] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } }
I20260806 11:41:56.209233 7762 leader_election.cc:288] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [CANDIDATE]: Term 1 election: Requested vote from peers b1deace4b7814a7ebe88c7d4d8c61454 (127.0.113.3:39351), 0342805529ae4c0aa7a5f1ada25c2449 (127.0.113.2:36693)
I20260806 11:41:56.209698 7454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f703801934844cb092911d766741b70a" candidate_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0342805529ae4c0aa7a5f1ada25c2449"
I20260806 11:41:56.209712 7657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f703801934844cb092911d766741b70a" candidate_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1deace4b7814a7ebe88c7d4d8c61454"
I20260806 11:41:56.209870 7657 raft_consensus.cc:3037] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:56.209877 7454 raft_consensus.cc:3037] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:56.211194 7454 raft_consensus.cc:2454] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3adfbbb6c0417aa331271e90a27b4c in term 1.
I20260806 11:41:56.211588 7170 leader_election.cc:302] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0342805529ae4c0aa7a5f1ada25c2449, ca3adfbbb6c0417aa331271e90a27b4c; no voters:
I20260806 11:41:56.211592 7657 raft_consensus.cc:2454] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3adfbbb6c0417aa331271e90a27b4c in term 1.
I20260806 11:41:56.211786 7762 raft_consensus.cc:2781] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:56.212057 7762 raft_consensus.cc:686] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 1 LEADER]: Becoming Leader. State: Replica: ca3adfbbb6c0417aa331271e90a27b4c, State: Running, Role: LEADER
I20260806 11:41:56.212442 7762 consensus_queue.cc:237] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } }
I20260806 11:41:56.215795 7056 catalog_manager.cc:5495] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c reported cstate change: term changed from 0 to 1, leader changed from <none> to ca3adfbbb6c0417aa331271e90a27b4c (127.0.113.1). New cstate: current_term: 1 leader_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3adfbbb6c0417aa331271e90a27b4c" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 44419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 } health_report { overall_health: UNKNOWN } } }
I20260806 11:41:56.241217 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.239497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.0.113.1:44419)
0806 11:41:56.239625 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.239629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.239642 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:41:56.239820 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.239823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.239883 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.239976 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.240788 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.240792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.240794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.240806 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.240833 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.240843 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.240950 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.241028 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.241069 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 11:41:56.246907 7454 raft_consensus.cc:1261] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [term 1 FOLLOWER]: Refusing update from remote peer ca3adfbbb6c0417aa331271e90a27b4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:56.246905 7657 raft_consensus.cc:1261] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [term 1 FOLLOWER]: Refusing update from remote peer ca3adfbbb6c0417aa331271e90a27b4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:41:56.247602 7767 consensus_queue.cc:1035] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1deace4b7814a7ebe88c7d4d8c61454" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:56.247810 7762 consensus_queue.cc:1035] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0342805529ae4c0aa7a5f1ada25c2449" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:41:56.257431 7771 mvcc.cc:205] Tried to move back new op lower bound from 7315523650540298240 to 7315523650409103360. Current Snapshot: MvccSnapshot[applied={T|T < 7315523650540298240}]
I20260806 11:41:56.266623 7058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:56.266798 7058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:41:56.269786 7058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9703d78042f9499dba1fe6cde9299f0e: Sending DeleteTablet for 3 replicas of tablet f703801934844cb092911d766741b70a
I20260806 11:41:56.270609 7234 tablet_service.cc:1514] Processing DeleteTablet for tablet f703801934844cb092911d766741b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:56 UTC) from {username='slave'} at 127.0.0.1:36812
I20260806 11:41:56.270879 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet f703801934844cb092911d766741b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:56 UTC) from {username='slave'} at 127.0.0.1:47980
I20260806 11:41:56.271106 7784 tablet_replica.cc:330] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c: stopping tablet replica
I20260806 11:41:56.271157 7627 tablet_service.cc:1514] Processing DeleteTablet for tablet f703801934844cb092911d766741b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:41:56 UTC) from {username='slave'} at 127.0.0.1:56550
I20260806 11:41:56.271397 7784 raft_consensus.cc:2229] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:41:56.271667 7786 tablet_replica.cc:330] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: stopping tablet replica
I20260806 11:41:56.271745 7784 raft_consensus.cc:2258] T f703801934844cb092911d766741b70a P ca3adfbbb6c0417aa331271e90a27b4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:56.272063 7786 raft_consensus.cc:2229] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:56.272152 7785 tablet_replica.cc:330] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449: stopping tablet replica
I20260806 11:41:56.272217 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 7144
I20260806 11:41:56.275954 7785 raft_consensus.cc:2229] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:41:56.276266 7786 raft_consensus.cc:2258] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:56.276266 7785 raft_consensus.cc:2258] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:41:56.278000 7786 ts_tablet_manager.cc:1905] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:41:56.283227 7785 ts_tablet_manager.cc:1905] T f703801934844cb092911d766741b70a P 0342805529ae4c0aa7a5f1ada25c2449: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 11:41:56.289683 7029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:44419 (error 108)
I20260806 11:41:56.289842 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 7342
I20260806 11:41:56.294198 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.293886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:44419 (local address 127.0.0.1:36840)
0806 11:41:56.294046 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:56.294116 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 11:41:56.294376 7029 catalog_manager.cc:4584] TS ca3adfbbb6c0417aa331271e90a27b4c (127.0.113.1:44419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f703801934844cb092911d766741b70a: Network error: Client connection negotiation failed: client connection to 127.0.113.1:44419: connect: Connection refused (error 111)
I20260806 11:41:56.304984 7786 ts_tablet_manager.cc:1918] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:41:56.305123 7786 log.cc:1196] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/f703801934844cb092911d766741b70a
I20260806 11:41:56.305524 7786 ts_tablet_manager.cc:1939] T f703801934844cb092911d766741b70a P b1deace4b7814a7ebe88c7d4d8c61454: Deleting consensus metadata
I20260806 11:41:56.311077 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 7532
I20260806 11:41:56.311564 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.311211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:36693 (local address 127.0.0.1:47984)
0806 11:41:56.311409 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:56.311464 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.2:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260806 11:41:56.328421 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 7005
2026-08-06T11:41:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1105 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 11:41:56.356980 452 mini_oidc.cc:152] Starting JWKS server
I20260806 11:41:56.358672 452 webserver.cc:460] Webserver started at https://127.0.0.1:35557/ using document root <none> and password file <none>
I20260806 11:41:56.358860 452 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 11:41:56.359140 452 webserver.cc:460] Webserver started at http://127.0.0.1:35219/ using document root <none> and password file <none>
2026-08-06T11:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:56Z Disabled control of system clock
I20260806 11:41:56.411103 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:34861
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:34861
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35557/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/testchainca.cert with env {}
W20260806 11:41:56.522511 7799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:56.522763 7799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:56.522800 7799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:56.526711 7799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:56.526760 7799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:56.526775 7799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:56.526791 7799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:56.526805 7799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 11:41:56.526834 7799 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 11:41:56.526849 7799 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35557/jwks.json
I20260806 11:41:56.532624 7799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:34861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:34861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35557/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:56.533558 7799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:56.534467 7799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:56.542948 7810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:56.542933 7812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:56.542935 7814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:56.543321 7799 server_base.cc:1034] running on GCE node
I20260806 11:41:56.543936 7799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:56.545164 7799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:56.546353 7799 hybrid_clock.cc:648] HybridClock initialized: now 1786016516546326 us; error 42 us; skew 500 ppm
I20260806 11:41:56.548218 7799 webserver.cc:460] Webserver started at http://127.0.113.62:37709/ using document root <none> and password file <none>
I20260806 11:41:56.548834 7799 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:56.548934 7799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:56.549166 7799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:56.551208 7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "2388fb88049b4f9488bc0199a1696a2f"
format_stamp: "Formatted at 2026-08-06 11:41:56 on dist-test-slave-45dx"
server_key: "e5b2b1729d3982d804767827614187c4"
server_key_iv: "2760e7af0ed6e6652acfc373db770423"
server_key_version: "encryptionkey@0"
I20260806 11:41:56.551819 7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "2388fb88049b4f9488bc0199a1696a2f"
format_stamp: "Formatted at 2026-08-06 11:41:56 on dist-test-slave-45dx"
server_key: "e5b2b1729d3982d804767827614187c4"
server_key_iv: "2760e7af0ed6e6652acfc373db770423"
server_key_version: "encryptionkey@0"
I20260806 11:41:56.555624 7799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260806 11:41:56.558579 7823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:56.559911 7799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 11:41:56.560045 7799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "2388fb88049b4f9488bc0199a1696a2f"
format_stamp: "Formatted at 2026-08-06 11:41:56 on dist-test-slave-45dx"
server_key: "e5b2b1729d3982d804767827614187c4"
server_key_iv: "2760e7af0ed6e6652acfc373db770423"
server_key_version: "encryptionkey@0"
I20260806 11:41:56.560161 7799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:56.576128 7799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:56.576818 7799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:56.577034 7799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:56.584039 7799 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:34861
I20260806 11:41:56.584074 7897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:34861 every 8 connection(s)
I20260806 11:41:56.585157 7799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:56.587064 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 7799
I20260806 11:41:56.587184 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:56.587507 452 external_mini_cluster.cc:1428] Setting key cf989b58b713a8f22e5c520d4b6badee
I20260806 11:41:56.589007 7898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:56.594573 7898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f: Bootstrap starting.
I20260806 11:41:56.597076 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.589023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.0.113.62:34861)
0806 11:41:56.589656 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.589671 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.589705 (+ 34us) server_negotiation.cc:407] Connection header received
0806 11:41:56.590332 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.590347 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.590714 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.591040 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.591487 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:56.592509 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:56.593451 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:56.593928 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:56.594114 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:56.595061 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.595068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.595082 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.595563 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.595635 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.595663 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.595799 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.595882 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.596150 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":142,"threads_started":1}
I20260806 11:41:56.597342 7898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:56.598207 7898 log.cc:824] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:56.600497 7898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f: No bootstrap required, opened a new log
I20260806 11:41:56.600505 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:41:56.604017 7898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2388fb88049b4f9488bc0199a1696a2f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34861 } }
I20260806 11:41:56.604259 7898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:56.604389 7898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2388fb88049b4f9488bc0199a1696a2f, State: Initialized, Role: FOLLOWER
I20260806 11:41:56.604890 7898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2388fb88049b4f9488bc0199a1696a2f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34861 } }
I20260806 11:41:56.605063 7898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:56.605147 7898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:56.605259 7898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:56.606141 7898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2388fb88049b4f9488bc0199a1696a2f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34861 } }
I20260806 11:41:56.606488 7898 leader_election.cc:302] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2388fb88049b4f9488bc0199a1696a2f; no voters:
I20260806 11:41:56.606801 7898 leader_election.cc:288] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:56.607256 7906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:56.607534 7906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [term 1 LEADER]: Becoming Leader. State: Replica: 2388fb88049b4f9488bc0199a1696a2f, State: Running, Role: LEADER
I20260806 11:41:56.607870 7906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2388fb88049b4f9488bc0199a1696a2f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34861 } }
I20260806 11:41:56.608026 7898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:56.609558 7909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2388fb88049b4f9488bc0199a1696a2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2388fb88049b4f9488bc0199a1696a2f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34861 } } }
I20260806 11:41:56.609695 7909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:56.609941 7905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2388fb88049b4f9488bc0199a1696a2f. Latest consensus state: current_term: 1 leader_uuid: "2388fb88049b4f9488bc0199a1696a2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2388fb88049b4f9488bc0199a1696a2f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34861 } } }
I20260806 11:41:56.610055 7905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f [sys.catalog]: This master's current role is: LEADER
W20260806 11:41:56.613015 7924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:41:56.613082 7924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:41:56.613164 7920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:56.613857 7920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:56.619488 7920 catalog_manager.cc:1347] Generated new cluster ID: 8a197ddc2d4548ccb22a64d5a8129099
I20260806 11:41:56.619556 7920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:56.633988 7920 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:56.635154 7920 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:56.641222 7920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2388fb88049b4f9488bc0199a1696a2f: Generated new TSK 0
I20260806 11:41:56.641978 7920 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 11:41:56.674947 7946 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 11:41:56.675064 7946 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 11:41:56.675606 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.650762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.0.113.62:34861)
0806 11:41:56.650896 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.650903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.651110 (+ 207us) server_negotiation.cc:407] Connection header received
0806 11:41:56.651328 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.651331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.651388 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.651468 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0806 11:41:56.651762 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:56.652252 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:56.653580 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:56.653788 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:56.653865 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:56.654195 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 11:41:56.675229 (+ 21034us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 11:41:56.675483 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51672: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 11:41:56.675710 7903 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51672: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 11:41:56.677237 7944 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:41:56.650662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:34861 (local address 127.0.0.1:51672)
0806 11:41:56.651026 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:56.651043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:56.651189 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:56.651513 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:56.651516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:56.651524 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 11:41:56.651652 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:56.651658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:56.652456 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:56.652460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:56.653462 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:56.653472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:56.653900 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:56.653904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:56.653994 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:56.654044 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 11:41:56.675462 (+ 21418us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 11:41:56.675702 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.113.62:34861: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":118,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.113.62:34861: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 11:41:56.678182 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 7799
2026-08-06T11:41:56Z chronyd exiting
I20260806 11:41:56.721187 452 test_util.cc:175] -----------------------------------------------
I20260806 11:41:56.721273 452 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786016488822243-452-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (366 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 11:41:56.722324 452 mini_oidc.cc:152] Starting JWKS server
I20260806 11:41:56.723230 452 webserver.cc:460] Webserver started at https://127.0.0.1:41069/ using document root <none> and password file <none>
I20260806 11:41:56.723313 452 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 11:41:56.723564 452 webserver.cc:460] Webserver started at http://127.0.0.1:36227/ using document root <none> and password file <none>
2026-08-06T11:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:56Z Disabled control of system clock
I20260806 11:41:56.733868 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:35317
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:36099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:35317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41069/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/testchainca.cert with env {}
W20260806 11:41:56.844806 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:56.845108 7957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:56.845188 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:56.849177 7957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:56.849254 7957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:56.849282 7957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:56.849318 7957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:56.849373 7957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 11:41:56.849413 7957 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 11:41:56.849447 7957 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41069/jwks.json
I20260806 11:41:56.854596 7957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:35317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41069/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:56.855613 7957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:56.856667 7957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:56.865362 7967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:56.865432 7964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:56.865535 7957 server_base.cc:1034] running on GCE node
W20260806 11:41:56.865432 7965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:56.866321 7957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:56.867416 7957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:56.868613 7957 hybrid_clock.cc:648] HybridClock initialized: now 1786016516868600 us; error 30 us; skew 500 ppm
I20260806 11:41:56.870498 7957 webserver.cc:460] Webserver started at http://127.0.113.62:44961/ using document root <none> and password file <none>
I20260806 11:41:56.871070 7957 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:56.871165 7957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:56.871394 7957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:56.873425 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "3ea3ee68a6564cbb929afd924828f26a"
format_stamp: "Formatted at 2026-08-06 11:41:56 on dist-test-slave-45dx"
server_key: "b9c0d38350c6bae21464c211c2e1ee5e"
server_key_iv: "e3aaf429d6ce740208f1b8a25bb37757"
server_key_version: "encryptionkey@0"
I20260806 11:41:56.873991 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "3ea3ee68a6564cbb929afd924828f26a"
format_stamp: "Formatted at 2026-08-06 11:41:56 on dist-test-slave-45dx"
server_key: "b9c0d38350c6bae21464c211c2e1ee5e"
server_key_iv: "e3aaf429d6ce740208f1b8a25bb37757"
server_key_version: "encryptionkey@0"
I20260806 11:41:56.878052 7957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:41:56.880568 7975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:56.881752 7957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:56.881964 7957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "3ea3ee68a6564cbb929afd924828f26a"
format_stamp: "Formatted at 2026-08-06 11:41:56 on dist-test-slave-45dx"
server_key: "b9c0d38350c6bae21464c211c2e1ee5e"
server_key_iv: "e3aaf429d6ce740208f1b8a25bb37757"
server_key_version: "encryptionkey@0"
I20260806 11:41:56.882083 7957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:56.898020 7957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:56.898650 7957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:56.898844 7957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:56.906574 7957 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:35317
I20260806 11:41:56.906653 8047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:35317 every 8 connection(s)
I20260806 11:41:56.907913 7957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:56.909552 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 7957
I20260806 11:41:56.909690 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:56.910002 452 external_mini_cluster.cc:1428] Setting key 93eaf9a97aec90c83e4ee83be8cbc474
I20260806 11:41:56.911260 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:56.916677 8048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a: Bootstrap starting.
I20260806 11:41:56.918958 8048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:56.919075 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:56.911328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.0.113.62:35317)
0806 11:41:56.911968 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:56.911983 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:56.912018 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:41:56.912716 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:56.912733 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:56.913108 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:56.913434 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:56.913892 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:56.914823 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:56.915771 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:56.916232 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:56.916455 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:56.917402 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:56.917407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:56.917419 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:56.917794 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:56.917846 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:56.917866 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:56.917982 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:56.918137 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 11:41:56.918381 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":150,"threads_started":1}
I20260806 11:41:56.919811 8048 log.cc:824] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:56.921867 8048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a: No bootstrap required, opened a new log
I20260806 11:41:56.921927 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:41:56.925542 8048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea3ee68a6564cbb929afd924828f26a" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35317 } }
I20260806 11:41:56.925770 8048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:56.925863 8048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea3ee68a6564cbb929afd924828f26a, State: Initialized, Role: FOLLOWER
I20260806 11:41:56.926263 8048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea3ee68a6564cbb929afd924828f26a" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35317 } }
I20260806 11:41:56.926421 8048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:56.926504 8048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:56.926621 8048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:57.003322 8048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea3ee68a6564cbb929afd924828f26a" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35317 } }
I20260806 11:41:57.003866 8048 leader_election.cc:302] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ea3ee68a6564cbb929afd924828f26a; no voters:
I20260806 11:41:57.004284 8048 leader_election.cc:288] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:57.004482 8057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:57.004778 8057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [term 1 LEADER]: Becoming Leader. State: Replica: 3ea3ee68a6564cbb929afd924828f26a, State: Running, Role: LEADER
I20260806 11:41:57.005184 8057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea3ee68a6564cbb929afd924828f26a" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35317 } }
I20260806 11:41:57.005789 8048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:57.007241 8059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ea3ee68a6564cbb929afd924828f26a. Latest consensus state: current_term: 1 leader_uuid: "3ea3ee68a6564cbb929afd924828f26a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea3ee68a6564cbb929afd924828f26a" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35317 } } }
I20260806 11:41:57.007387 8059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:57.007717 8058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ea3ee68a6564cbb929afd924828f26a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea3ee68a6564cbb929afd924828f26a" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35317 } } }
I20260806 11:41:57.007830 8058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a [sys.catalog]: This master's current role is: LEADER
W20260806 11:41:57.010914 8079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:41:57.011018 8079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:41:57.011104 8070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:57.012044 8070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:57.017652 8070 catalog_manager.cc:1347] Generated new cluster ID: 4b18c32f6108428c98fb2803663b5fde
I20260806 11:41:57.017726 8070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:57.031174 8070 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:57.032531 8070 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:57.043771 8070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ea3ee68a6564cbb929afd924828f26a: Generated new TSK 0
I20260806 11:41:57.044530 8070 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 11:41:57.082310 8092 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 11:41:57.082410 8092 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 11:41:57.083060 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:57.065911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.0.113.62:35317)
0806 11:41:57.066036 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:57.066043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:57.066223 (+ 180us) server_negotiation.cc:407] Connection header received
0806 11:41:57.066378 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:57.066381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:57.066438 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:57.066525 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0806 11:41:57.066778 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.067339 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.068593 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.068931 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.069013 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:57.069393 (+ 380us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 11:41:57.082673 (+ 13280us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 11:41:57.082912 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 11:41:57.083132 8052 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 11:41:57.083160 8091 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:41:57.065763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35317 (local address 127.0.0.1:59922)
0806 11:41:57.066130 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:57.066145 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:57.066260 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:57.066550 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:57.066552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:57.066561 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 11:41:57.066677 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:57.066683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.067448 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.067452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:57.068481 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:57.068492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.069046 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.069050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:57.069162 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:57.069190 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 11:41:57.082850 (+ 13660us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 11:41:57.082997 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.113.62:35317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260806 11:41:57.083791 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 7957
2026-08-06T11:41:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (410 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 11:41:57.132833 452 mini_oidc.cc:152] Starting JWKS server
I20260806 11:41:57.133701 452 webserver.cc:460] Webserver started at https://127.0.0.1:41917/ using document root <none> and password file <none>
I20260806 11:41:57.133818 452 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 11:41:57.134095 452 webserver.cc:460] Webserver started at http://127.0.0.1:44745/ using document root <none> and password file <none>
2026-08-06T11:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:57Z Disabled control of system clock
I20260806 11:41:57.146169 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33709
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:33709
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41917/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 11:41:57.277993 8101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:57.278261 8101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:57.278301 8101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:57.282419 8101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:57.282480 8101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:57.282495 8101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:57.282510 8101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:57.282524 8101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 11:41:57.282585 8101 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 11:41:57.282611 8101 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41917/jwks.json
I20260806 11:41:57.287364 8101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:33709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41917/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:57.288439 8101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:57.289865 8101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:57.298750 8111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:57.298772 8113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:57.298805 8101 server_base.cc:1034] running on GCE node
W20260806 11:41:57.298799 8109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:57.299628 8101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:57.300758 8101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:57.301951 8101 hybrid_clock.cc:648] HybridClock initialized: now 1786016517301926 us; error 45 us; skew 500 ppm
I20260806 11:41:57.303925 8101 webserver.cc:460] Webserver started at http://127.0.113.62:35585/ using document root <none> and password file <none>
I20260806 11:41:57.304597 8101 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:57.304695 8101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:57.304937 8101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:57.306933 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "57a779c5de254458b71a4e0cdf19b3fe"
format_stamp: "Formatted at 2026-08-06 11:41:57 on dist-test-slave-45dx"
server_key: "50749d9a0632ea40b20e25775b860b43"
server_key_iv: "9dd68a5868ad3f06a7ebbab698d297a6"
server_key_version: "encryptionkey@0"
I20260806 11:41:57.307546 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "57a779c5de254458b71a4e0cdf19b3fe"
format_stamp: "Formatted at 2026-08-06 11:41:57 on dist-test-slave-45dx"
server_key: "50749d9a0632ea40b20e25775b860b43"
server_key_iv: "9dd68a5868ad3f06a7ebbab698d297a6"
server_key_version: "encryptionkey@0"
I20260806 11:41:57.311558 8101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 11:41:57.314039 8120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:57.315201 8101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:41:57.315300 8101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "57a779c5de254458b71a4e0cdf19b3fe"
format_stamp: "Formatted at 2026-08-06 11:41:57 on dist-test-slave-45dx"
server_key: "50749d9a0632ea40b20e25775b860b43"
server_key_iv: "9dd68a5868ad3f06a7ebbab698d297a6"
server_key_version: "encryptionkey@0"
I20260806 11:41:57.315394 8101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:57.329978 8101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:57.330582 8101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:57.330734 8101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:57.337386 8101 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:33709
I20260806 11:41:57.337409 8195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:33709 every 8 connection(s)
I20260806 11:41:57.338295 8101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:57.341969 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:57.343959 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 8101
I20260806 11:41:57.344082 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:57.344414 452 external_mini_cluster.cc:1428] Setting key 7a5eb7b02c18c06a98240f5d71ac2169
I20260806 11:41:57.347940 8196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe: Bootstrap starting.
I20260806 11:41:57.350116 8196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:57.350829 8196 log.cc:824] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:57.353150 8196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe: No bootstrap required, opened a new log
I20260806 11:41:57.353729 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:57.345750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.0.113.62:33709)
0806 11:41:57.346348 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:57.346362 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:57.346397 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:41:57.347131 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:57.347149 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:57.347515 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:57.347838 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:57.348282 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.349291 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.350186 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.350668 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.350842 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:57.351714 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:57.351723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:57.351741 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:57.352211 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:57.352285 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:57.352315 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:57.352477 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:57.352575 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 11:41:57.352822 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260806 11:41:57.356352 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:41:57.356833 8196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a779c5de254458b71a4e0cdf19b3fe" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33709 } }
I20260806 11:41:57.357115 8196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:57.357180 8196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a779c5de254458b71a4e0cdf19b3fe, State: Initialized, Role: FOLLOWER
I20260806 11:41:57.357648 8196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a779c5de254458b71a4e0cdf19b3fe" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33709 } }
I20260806 11:41:57.357805 8196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:57.357887 8196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:57.358011 8196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:57.358876 8196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a779c5de254458b71a4e0cdf19b3fe" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33709 } }
I20260806 11:41:57.359220 8196 leader_election.cc:302] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57a779c5de254458b71a4e0cdf19b3fe; no voters:
I20260806 11:41:57.359517 8196 leader_election.cc:288] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:57.359632 8203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:57.359899 8203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [term 1 LEADER]: Becoming Leader. State: Replica: 57a779c5de254458b71a4e0cdf19b3fe, State: Running, Role: LEADER
I20260806 11:41:57.360193 8203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a779c5de254458b71a4e0cdf19b3fe" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33709 } }
I20260806 11:41:57.360693 8196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:57.361892 8204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57a779c5de254458b71a4e0cdf19b3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a779c5de254458b71a4e0cdf19b3fe" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33709 } } }
I20260806 11:41:57.361884 8205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57a779c5de254458b71a4e0cdf19b3fe. Latest consensus state: current_term: 1 leader_uuid: "57a779c5de254458b71a4e0cdf19b3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a779c5de254458b71a4e0cdf19b3fe" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33709 } } }
I20260806 11:41:57.362035 8204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:57.362057 8205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe [sys.catalog]: This master's current role is: LEADER
W20260806 11:41:57.365797 8224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:41:57.365862 8224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:41:57.365942 8213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:57.366616 8213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:57.371927 8213 catalog_manager.cc:1347] Generated new cluster ID: 8de31f337e7d4d32aa378a404a050444
I20260806 11:41:57.371987 8213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:57.386670 8213 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:57.387518 8213 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:57.394117 8213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57a779c5de254458b71a4e0cdf19b3fe: Generated new TSK 0
I20260806 11:41:57.394770 8213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:41:57.414199 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:57.409017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.0.113.62:33709)
0806 11:41:57.409163 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:57.409168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:57.409358 (+ 190us) server_negotiation.cc:407] Connection header received
0806 11:41:57.409492 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:57.409495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:57.409537 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:57.409619 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0806 11:41:57.409895 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.410450 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.411822 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.412036 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.412113 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:57.412461 (+ 348us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 11:41:57.413827 (+ 1366us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 11:41:57.414050 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 11:41:57.414263 8201 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 11:41:57.416263 8232 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:41:57.408933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33709 (local address 127.0.0.1:37420)
0806 11:41:57.409263 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:57.409278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:57.409392 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:57.409645 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:57.409648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:57.409655 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0806 11:41:57.409769 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:57.409777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.410559 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.410563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:57.411693 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:57.411705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.412154 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.412158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:57.412246 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:57.412274 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 11:41:57.413990 (+ 1716us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 11:41:57.416087 (+ 2097us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.113.62:33709: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":1834,"thread_start_us":123,"threads_started":1}
I20260806 11:41:57.416957 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 8101
E20260806 11:41:57.427340 8233 webserver.cc:533] Webserver: sslize failed:
E20260806 11:41:57.427454 8233 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T11:41:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T11:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:57Z Disabled control of system clock
I20260806 11:41:57.473693 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:34715
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:34715
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/testchainca.cert with env {}
W20260806 11:41:57.586210 8238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:57.586472 8238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:57.586509 8238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:57.590503 8238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:57.590557 8238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:57.590572 8238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:57.590590 8238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:57.590605 8238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 11:41:57.590619 8238 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 11:41:57.590632 8238 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 11:41:57.595775 8238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:34715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:34715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:57.596796 8238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:57.597749 8238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:57.606424 8250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:57.606470 8248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:57.606470 8246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:57.606624 8238 server_base.cc:1034] running on GCE node
I20260806 11:41:57.607524 8238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:57.608727 8238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:57.609906 8238 hybrid_clock.cc:648] HybridClock initialized: now 1786016517609877 us; error 41 us; skew 500 ppm
I20260806 11:41:57.611763 8238 webserver.cc:460] Webserver started at http://127.0.113.62:46535/ using document root <none> and password file <none>
I20260806 11:41:57.612422 8238 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:57.612494 8238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:57.612731 8238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:57.614826 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "194af0c5856b41718d8a5ae7d74b11bf"
format_stamp: "Formatted at 2026-08-06 11:41:57 on dist-test-slave-45dx"
server_key: "f72855c9720d21873dae8612c53e2320"
server_key_iv: "303b992086d903bc5b442d325e8314fc"
server_key_version: "encryptionkey@0"
I20260806 11:41:57.615432 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "194af0c5856b41718d8a5ae7d74b11bf"
format_stamp: "Formatted at 2026-08-06 11:41:57 on dist-test-slave-45dx"
server_key: "f72855c9720d21873dae8612c53e2320"
server_key_iv: "303b992086d903bc5b442d325e8314fc"
server_key_version: "encryptionkey@0"
I20260806 11:41:57.619077 8238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:41:57.621593 8260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:57.622859 8238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:41:57.622967 8238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "194af0c5856b41718d8a5ae7d74b11bf"
format_stamp: "Formatted at 2026-08-06 11:41:57 on dist-test-slave-45dx"
server_key: "f72855c9720d21873dae8612c53e2320"
server_key_iv: "303b992086d903bc5b442d325e8314fc"
server_key_version: "encryptionkey@0"
I20260806 11:41:57.623116 8238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:57.639423 8238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:57.640060 8238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:57.640257 8238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:57.647145 8238 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:34715
I20260806 11:41:57.647173 8342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:34715 every 8 connection(s)
I20260806 11:41:57.648052 8238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:57.649474 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 8238
I20260806 11:41:57.649608 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:57.649935 452 external_mini_cluster.cc:1428] Setting key dd027fe358270bad1784ac38ef14090a
I20260806 11:41:57.651746 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:57.657964 8343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf: Bootstrap starting.
I20260806 11:41:57.659111 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:57.651308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.0.113.62:34715)
0806 11:41:57.652011 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:57.652028 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:57.652063 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:41:57.652732 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:57.652749 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:57.653141 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:57.653450 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:57.653813 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.654726 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.655883 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.656381 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.656496 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:57.657358 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:57.657363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:57.657374 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:41:57.657731 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:57.657782 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:41:57.657812 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:57.657884 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:57.658093 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 11:41:57.658452 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
I20260806 11:41:57.660192 8343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:57.661055 8343 log.cc:824] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:57.661954 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:41:57.663121 8343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf: No bootstrap required, opened a new log
I20260806 11:41:57.666479 8343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194af0c5856b41718d8a5ae7d74b11bf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34715 } }
I20260806 11:41:57.666710 8343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:57.666787 8343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194af0c5856b41718d8a5ae7d74b11bf, State: Initialized, Role: FOLLOWER
I20260806 11:41:57.667229 8343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194af0c5856b41718d8a5ae7d74b11bf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34715 } }
I20260806 11:41:57.667390 8343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:57.667475 8343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:57.667572 8343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:57.668468 8343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194af0c5856b41718d8a5ae7d74b11bf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34715 } }
I20260806 11:41:57.668787 8343 leader_election.cc:302] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 194af0c5856b41718d8a5ae7d74b11bf; no voters:
I20260806 11:41:57.669140 8343 leader_election.cc:288] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:57.669180 8350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:57.669428 8350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [term 1 LEADER]: Becoming Leader. State: Replica: 194af0c5856b41718d8a5ae7d74b11bf, State: Running, Role: LEADER
I20260806 11:41:57.669745 8350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194af0c5856b41718d8a5ae7d74b11bf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34715 } }
I20260806 11:41:57.670365 8343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:57.671501 8354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 194af0c5856b41718d8a5ae7d74b11bf. Latest consensus state: current_term: 1 leader_uuid: "194af0c5856b41718d8a5ae7d74b11bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194af0c5856b41718d8a5ae7d74b11bf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34715 } } }
I20260806 11:41:57.671643 8354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:57.671926 8351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "194af0c5856b41718d8a5ae7d74b11bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194af0c5856b41718d8a5ae7d74b11bf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34715 } } }
I20260806 11:41:57.672024 8351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:57.672132 8361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:57.675110 8361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:57.681176 8361 catalog_manager.cc:1347] Generated new cluster ID: 107f1848f9e3444ebc3faabf1eef5b0d
I20260806 11:41:57.681245 8361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:57.699149 8361 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:57.700079 8361 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:57.716905 8361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 194af0c5856b41718d8a5ae7d74b11bf: Generated new TSK 0
I20260806 11:41:57.717589 8361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:41:57.790776 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:57.728916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.0.113.62:34715)
0806 11:41:57.729055 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:57.729059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:57.729264 (+ 205us) server_negotiation.cc:407] Connection header received
0806 11:41:57.729434 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:57.729436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:57.729481 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:57.729562 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0806 11:41:57.729849 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.730383 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.731615 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.731834 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.731915 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:57.732244 (+ 329us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 11:41:57.790393 (+ 58149us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 11:41:57.790652 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36770: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 11:41:57.790874 8348 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36770: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 11:41:57.791402 8378 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:41:57.728855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:34715 (local address 127.0.0.1:36770)
0806 11:41:57.729169 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:57.729184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:57.729298 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:57.729588 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:57.729592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:57.729600 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 11:41:57.729714 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:57.729720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.730501 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.730505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:57.731484 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:57.731495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:57.731964 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:57.731968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:57.732061 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:57.732088 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 11:41:57.790670 (+ 58582us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 11:41:57.790910 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.113.62:34715: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":125,"threads_started":1}
I20260806 11:41:57.792223 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 8238
2026-08-06T11:41:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (362 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:41:58 dist-test-slave-45dx krb5kdc[8396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:41:58 dist-test-slave-45dx krb5kdc[8396](info): set up 2 sockets
Aug 06 11:41:58 dist-test-slave-45dx krb5kdc[8396](info): commencing operation
krb5kdc: starting...
W20260806 11:41:59.301420 452 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:41:59 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:41:59.425428 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:40479
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:40479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:59.538252 8414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:59.538604 8414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:59.538704 8414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:59.543257 8414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:41:59.543356 8414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:59.543393 8414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:59.543434 8414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:41:59.543473 8414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:41:59.549041 8414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:40479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:40479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:59.550181 8414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:59.551303 8414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:59.560089 8422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:59.560099 8423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:59.560168 8414 server_base.cc:1034] running on GCE node
W20260806 11:41:59.560081 8425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:59.561008 8414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:59.562211 8414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:59.563406 8414 hybrid_clock.cc:648] HybridClock initialized: now 1786016519563394 us; error 26 us; skew 500 ppm
Aug 06 11:41:59 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:59.566321 8414 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:41:59.567195 8414 webserver.cc:460] Webserver started at http://127.0.113.62:33271/ using document root <none> and password file <none>
I20260806 11:41:59.567762 8414 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:59.567854 8414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:59.568091 8414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:59.570173 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "57dd26ac75cb44fa9bb62c71b4157334"
format_stamp: "Formatted at 2026-08-06 11:41:59 on dist-test-slave-45dx"
server_key: "3e4119b89751e8b365a51aa1d5a69246"
server_key_iv: "791d108d5afab615da93fe98b5df1dcf"
server_key_version: "encryptionkey@0"
I20260806 11:41:59.570755 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "57dd26ac75cb44fa9bb62c71b4157334"
format_stamp: "Formatted at 2026-08-06 11:41:59 on dist-test-slave-45dx"
server_key: "3e4119b89751e8b365a51aa1d5a69246"
server_key_iv: "791d108d5afab615da93fe98b5df1dcf"
server_key_version: "encryptionkey@0"
I20260806 11:41:59.574532 8414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:41:59.576949 8432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:59.578083 8414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:59.578222 8414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "57dd26ac75cb44fa9bb62c71b4157334"
format_stamp: "Formatted at 2026-08-06 11:41:59 on dist-test-slave-45dx"
server_key: "3e4119b89751e8b365a51aa1d5a69246"
server_key_iv: "791d108d5afab615da93fe98b5df1dcf"
server_key_version: "encryptionkey@0"
I20260806 11:41:59.578333 8414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:59.602196 8414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:59.605500 8414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:59.605698 8414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:59.612335 8414 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:40479
I20260806 11:41:59.612665 8519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:40479 every 8 connection(s)
I20260806 11:41:59.613304 8414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:41:59.616025 8520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:41:59.621354 8520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334: Bootstrap starting.
I20260806 11:41:59.622179 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 8414
I20260806 11:41:59.622305 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:41:59.622646 452 external_mini_cluster.cc:1428] Setting key 146b3392bd7bc2994f8f308bff8cb86c
I20260806 11:41:59.624265 8520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334: Neither blocks nor log segments found. Creating new log.
I20260806 11:41:59.625017 8520 log.cc:824] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334: Log is configured to *not* fsync() on all Append() calls
I20260806 11:41:59.627157 8520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334: No bootstrap required, opened a new log
Aug 06 11:41:59 dist-test-slave-45dx krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:59.631564 8520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dd26ac75cb44fa9bb62c71b4157334" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40479 } }
I20260806 11:41:59.631839 8520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:41:59.631920 8520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57dd26ac75cb44fa9bb62c71b4157334, State: Initialized, Role: FOLLOWER
I20260806 11:41:59.632450 8520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dd26ac75cb44fa9bb62c71b4157334" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40479 } }
I20260806 11:41:59.632635 8520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:41:59.632709 8520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:41:59.632803 8520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:41:59.633998 8520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dd26ac75cb44fa9bb62c71b4157334" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40479 } }
I20260806 11:41:59.634361 8520 leader_election.cc:302] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57dd26ac75cb44fa9bb62c71b4157334; no voters:
I20260806 11:41:59.634707 8520 leader_election.cc:288] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:41:59.634812 8530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:41:59.635039 8530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [term 1 LEADER]: Becoming Leader. State: Replica: 57dd26ac75cb44fa9bb62c71b4157334, State: Running, Role: LEADER
I20260806 11:41:59.635426 8530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dd26ac75cb44fa9bb62c71b4157334" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40479 } }
I20260806 11:41:59.635906 8520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:41:59.637172 8531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57dd26ac75cb44fa9bb62c71b4157334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dd26ac75cb44fa9bb62c71b4157334" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40479 } } }
I20260806 11:41:59.637292 8531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:59.637625 8532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57dd26ac75cb44fa9bb62c71b4157334. Latest consensus state: current_term: 1 leader_uuid: "57dd26ac75cb44fa9bb62c71b4157334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dd26ac75cb44fa9bb62c71b4157334" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40479 } } }
I20260806 11:41:59.637715 8532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334 [sys.catalog]: This master's current role is: LEADER
I20260806 11:41:59.637850 8539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:41:59.638374 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:59.623802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.0.113.62:40479)
0806 11:41:59.624215 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:59.624229 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:59.624264 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:41:59.624963 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:59.624979 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:59.625346 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:59.625717 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:59.626634 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:59.627630 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:59.628796 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:59.629255 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:59.629367 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:59.632243 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:59.632249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:59.632264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:59.632293 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:59.634343 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:59.634911 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:59.634917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:59.634922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:59.635077 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:59.635434 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:59.635439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:59.635444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:59.635955 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:59.636189 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:59.636629 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 11:41:59.637159 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
I20260806 11:41:59.641350 8539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:41:59.646539 8539 catalog_manager.cc:1347] Generated new cluster ID: afb7bc4ccaa84eccbc7d7ff4bd9b3e67
I20260806 11:41:59.646653 8539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:41:59.666129 8539 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:41:59.667315 8539 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:41:59.672655 8539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57dd26ac75cb44fa9bb62c71b4157334: Generated new TSK 0
I20260806 11:41:59.673400 8539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:41:59.709237 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40479
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:41:59.818720 8555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:41:59.818961 8555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:41:59.819001 8555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:41:59.823339 8555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:41:59.823392 8555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:41:59.823447 8555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:41:59.828405 8555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:41:59.829257 8555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:41:59.830183 8555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:41:59.839830 8563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:59.839843 8565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:41:59.839843 8560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:41:59.840073 8555 server_base.cc:1034] running on GCE node
I20260806 11:41:59.840649 8555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:41:59.841197 8555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:41:59.842362 8555 hybrid_clock.cc:648] HybridClock initialized: now 1786016519842351 us; error 27 us; skew 500 ppm
Aug 06 11:41:59 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:41:59.848579 8555 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:41:59.849439 8555 webserver.cc:460] Webserver started at http://127.0.113.1:42813/ using document root <none> and password file <none>
I20260806 11:41:59.849984 8555 fs_manager.cc:359] Metadata directory not provided
I20260806 11:41:59.850077 8555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:41:59.850307 8555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:41:59.852370 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "fd20ab753dbb49f691aa7ae6254e6ac0"
format_stamp: "Formatted at 2026-08-06 11:41:59 on dist-test-slave-45dx"
server_key: "3ffa8d6f67853f95320e47920620a2d1"
server_key_iv: "99b7459a8cae3837d255f7b87e2a7f7c"
server_key_version: "encryptionkey@0"
I20260806 11:41:59.852959 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "fd20ab753dbb49f691aa7ae6254e6ac0"
format_stamp: "Formatted at 2026-08-06 11:41:59 on dist-test-slave-45dx"
server_key: "3ffa8d6f67853f95320e47920620a2d1"
server_key_iv: "99b7459a8cae3837d255f7b87e2a7f7c"
server_key_version: "encryptionkey@0"
I20260806 11:41:59.856588 8555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:41:59.859113 8576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:59.860234 8555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:41:59.860390 8555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "fd20ab753dbb49f691aa7ae6254e6ac0"
format_stamp: "Formatted at 2026-08-06 11:41:59 on dist-test-slave-45dx"
server_key: "3ffa8d6f67853f95320e47920620a2d1"
server_key_iv: "99b7459a8cae3837d255f7b87e2a7f7c"
server_key_version: "encryptionkey@0"
I20260806 11:41:59.860500 8555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:41:59.882817 8555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:41:59.885896 8555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:41:59.886091 8555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:41:59.886644 8555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:41:59.887436 8555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:41:59.887506 8555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:59.887575 8555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:41:59.887626 8555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:41:59.896857 8555 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:36191
I20260806 11:41:59.896888 8740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:36191 every 8 connection(s)
I20260806 11:41:59.897749 8555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:41:59.905580 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 8555
I20260806 11:41:59.905725 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:41:59.906087 452 external_mini_cluster.cc:1428] Setting key 15d0a7454daf15bf18246db82c0a88fb
Aug 06 11:41:59 dist-test-slave-45dx krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:41:59.911479 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:59.899585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:33893 (local address 127.0.113.62:40479)
0806 11:41:59.899726 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 11:41:59.899730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:41:59.900570 (+ 840us) server_negotiation.cc:407] Connection header received
0806 11:41:59.901887 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:41:59.901889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:41:59.901938 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:41:59.902016 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:41:59.903127 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:59.903658 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:59.904755 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:41:59.905003 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:41:59.905093 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:59.907850 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:41:59.907854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:41:59.907856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:41:59.907880 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:41:59.909412 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:59.909940 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:59.909944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:59.909946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:59.910034 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:41:59.910398 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:41:59.910402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:41:59.910404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:41:59.910584 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:41:59.910715 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 11:41:59.911185 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 11:41:59.911310 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:41:59.912220 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:41:59.899924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40479 (local address 127.0.113.1:33893)
0806 11:41:59.900405 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 11:41:59.900447 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 11:41:59.901681 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:41:59.902171 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:41:59.902177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:41:59.902513 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:41:59.902982 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:59.902995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:59.903768 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:59.903770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:59.904640 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:41:59.904650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:41:59.905139 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:41:59.905142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:41:59.905270 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:41:59.906029 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:41:59.906050 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:41:59.907675 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:41:59.909539 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:59.909545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:59.909557 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:59.909829 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:59.910133 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:41:59.910137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:41:59.910139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:41:59.910290 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:41:59.910715 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:41:59.910722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:41:59.910952 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 11:41:59.911141 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 11:41:59.911452 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
I20260806 11:41:59.913503 8741 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40479
I20260806 11:41:59.913745 8741 heartbeater.cc:464] Registering TS with master...
I20260806 11:41:59.914238 8741 heartbeater.cc:511] Master 127.0.113.62:40479 requested a full tablet report, sending...
I20260806 11:41:59.915886 8456 ts_manager.cc:194] Registered new tserver with Master: fd20ab753dbb49f691aa7ae6254e6ac0 (127.0.113.1:36191)
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:41:59.917248 8456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:33893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:41:59.933586 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40479
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:00.043653 8746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:00.043910 8746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:00.043954 8746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:00.047823 8746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:00.047883 8746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:00.047940 8746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:42:00.053037 8746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:00.053975 8746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:00.054940 8746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:00.064432 8755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:00.064438 8754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:00.064438 8758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:00.064719 8746 server_base.cc:1034] running on GCE node
I20260806 11:42:00.065199 8746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:00.065835 8746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:00.067028 8746 hybrid_clock.cc:648] HybridClock initialized: now 1786016520067020 us; error 36 us; skew 500 ppm
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:00.070020 8746 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:42:00.070997 8746 webserver.cc:460] Webserver started at http://127.0.113.2:38431/ using document root <none> and password file <none>
I20260806 11:42:00.071599 8746 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:00.071693 8746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:00.071959 8746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:00.074131 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "8af7467c121d42de951fd0e93baf8206"
format_stamp: "Formatted at 2026-08-06 11:42:00 on dist-test-slave-45dx"
server_key: "440f9bb96f33724e205515c00130d538"
server_key_iv: "3473b15f8631bbbbf560e3cb365ea195"
server_key_version: "encryptionkey@0"
I20260806 11:42:00.074713 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "8af7467c121d42de951fd0e93baf8206"
format_stamp: "Formatted at 2026-08-06 11:42:00 on dist-test-slave-45dx"
server_key: "440f9bb96f33724e205515c00130d538"
server_key_iv: "3473b15f8631bbbbf560e3cb365ea195"
server_key_version: "encryptionkey@0"
I20260806 11:42:00.078630 8746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 11:42:00.081435 8772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:00.082850 8746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:42:00.082947 8746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "8af7467c121d42de951fd0e93baf8206"
format_stamp: "Formatted at 2026-08-06 11:42:00 on dist-test-slave-45dx"
server_key: "440f9bb96f33724e205515c00130d538"
server_key_iv: "3473b15f8631bbbbf560e3cb365ea195"
server_key_version: "encryptionkey@0"
I20260806 11:42:00.083094 8746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:00.099752 8746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:00.102744 8746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:00.102952 8746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:00.103524 8746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:00.104415 8746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:00.104465 8746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:00.104529 8746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:00.104568 8746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:00.113804 8746 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:36583
I20260806 11:42:00.113813 8928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:36583 every 8 connection(s)
I20260806 11:42:00.114724 8746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:42:00.120445 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 8746
I20260806 11:42:00.120571 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:42:00.120858 452 external_mini_cluster.cc:1428] Setting key 6e25b193451958640a7f3fea2b1aff12
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:42:00.129565 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:00.116667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:60543 (local address 127.0.113.62:40479)
0806 11:42:00.116812 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:00.116859 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:00.117746 (+ 887us) server_negotiation.cc:407] Connection header received
0806 11:42:00.119175 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:00.119179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:00.119236 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:00.119353 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:00.120680 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.121239 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.122643 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.122857 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.122966 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:00.125466 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:00.125476 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:00.125478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:00.125501 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:00.127346 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.127868 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.127872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.127875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.127979 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.128342 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.128348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.128353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.128621 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:00.128745 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:00.129208 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 11:42:00.129360 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:42:00.130216 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:00.117043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40479 (local address 127.0.113.2:60543)
0806 11:42:00.117499 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:00.117543 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:00.118972 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:00.119517 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:00.119524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:00.119874 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:00.120500 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:00.120525 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.121371 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.121377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:00.122495 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:00.122506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.123003 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.123008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:00.123175 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:00.123785 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:00.123801 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:00.125291 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:00.127472 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:00.127476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:00.127486 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:00.127760 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:00.128079 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:00.128081 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:00.128083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:00.128213 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:00.128733 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:00.128739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:00.128961 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 11:42:00.129116 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 11:42:00.129363 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260806 11:42:00.131429 8929 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40479
I20260806 11:42:00.131672 8929 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:42:00.132177 8929 heartbeater.cc:511] Master 127.0.113.62:40479 requested a full tablet report, sending...
I20260806 11:42:00.133255 8456 ts_manager.cc:194] Registered new tserver with Master: 8af7467c121d42de951fd0e93baf8206 (127.0.113.2:36583)
I20260806 11:42:00.133831 8456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:60543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:42:00.151386 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40479
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:00.266589 8934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:00.266932 8934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:00.267004 8934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:00.272423 8934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:00.272475 8934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:00.272567 8934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:42:00.277717 8934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:00.278651 8934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:00.279606 8934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:00.289510 8945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:00.289601 8944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:00.289630 8934 server_base.cc:1034] running on GCE node
W20260806 11:42:00.289526 8947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:00.290196 8934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:00.290804 8934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:00.291985 8934 hybrid_clock.cc:648] HybridClock initialized: now 1786016520291957 us; error 38 us; skew 500 ppm
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:00.294615 8934 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:42:00.295433 8934 webserver.cc:460] Webserver started at http://127.0.113.3:39689/ using document root <none> and password file <none>
I20260806 11:42:00.295965 8934 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:00.296051 8934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:00.296273 8934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:00.298231 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "83482684f82f4ef388e03768137545fe"
format_stamp: "Formatted at 2026-08-06 11:42:00 on dist-test-slave-45dx"
server_key: "81b5c1de27611e5340d1fc51f047f4db"
server_key_iv: "bb6b8dd9c3181c89fd426717865cf782"
server_key_version: "encryptionkey@0"
I20260806 11:42:00.298779 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "83482684f82f4ef388e03768137545fe"
format_stamp: "Formatted at 2026-08-06 11:42:00 on dist-test-slave-45dx"
server_key: "81b5c1de27611e5340d1fc51f047f4db"
server_key_iv: "bb6b8dd9c3181c89fd426717865cf782"
server_key_version: "encryptionkey@0"
I20260806 11:42:00.302377 8934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 11:42:00.304731 8959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:00.305907 8934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:42:00.306046 8934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "83482684f82f4ef388e03768137545fe"
format_stamp: "Formatted at 2026-08-06 11:42:00 on dist-test-slave-45dx"
server_key: "81b5c1de27611e5340d1fc51f047f4db"
server_key_iv: "bb6b8dd9c3181c89fd426717865cf782"
server_key_version: "encryptionkey@0"
I20260806 11:42:00.306150 8934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:00.338388 8934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:00.341559 8934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:00.341769 8934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:00.342398 8934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:00.343254 8934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:00.343343 8934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:00.343413 8934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:00.343457 8934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:00.352566 8934 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:40481
I20260806 11:42:00.352604 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:40481 every 8 connection(s)
I20260806 11:42:00.353438 8934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:42:00.358485 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 8934
I20260806 11:42:00.358629 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:42:00.358968 452 external_mini_cluster.cc:1428] Setting key ab9febf40d4b34796afbd67bda6ddef1
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:42:00.367055 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:00.355196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:43739 (local address 127.0.113.62:40479)
0806 11:42:00.355332 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:00.355336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:00.356119 (+ 783us) server_negotiation.cc:407] Connection header received
0806 11:42:00.357678 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:00.357681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:00.357741 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:00.357851 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:00.359134 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.359780 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.361084 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.361345 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.361423 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:00.363722 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:00.363725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:00.363727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:00.363751 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:00.365188 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.365649 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.365653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.365655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.365755 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.366099 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.366101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.366102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.366295 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:00.366377 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:00.366847 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 11:42:00.366941 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 11:42:00.367813 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:00.355532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40479 (local address 127.0.113.3:43739)
0806 11:42:00.355974 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:00.356018 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:00.357454 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:00.358035 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:00.358041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:00.358401 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:00.358970 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:00.358983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.359932 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.359935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:00.360949 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:00.360964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.361473 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.361476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:00.361600 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:00.362100 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:00.362115 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:00.363536 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:00.365294 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:00.365299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:00.365308 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:00.365553 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:00.365843 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:00.365845 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:00.365847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:00.365988 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:00.366387 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:00.366392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:00.366594 (+ 202us) client_negotiation.cc:769] Sending connection context
0806 11:42:00.366747 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 11:42:00.367010 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260806 11:42:00.368980 9133 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40479
I20260806 11:42:00.369179 9133 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:00.369568 9133 heartbeater.cc:511] Master 127.0.113.62:40479 requested a full tablet report, sending...
I20260806 11:42:00.370496 8456 ts_manager.cc:194] Registered new tserver with Master: 83482684f82f4ef388e03768137545fe (127.0.113.3:40481)
I20260806 11:42:00.370990 8456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:43739
I20260806 11:42:00.372802 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/test-admin.keytab.
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:00.389290 452 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 11:42:00.391901 9144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:00.390605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.0.113.1:36191)
0806 11:42:00.390895 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:00.390913 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:00.390946 (+ 33us) server_negotiation.cc:407] Connection header received
0806 11:42:00.390994 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:00.390999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:00.391162 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:00.391282 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:00.391693 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43272: BlockingRecv error: recv got EOF from 127.0.0.1:43272 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":127,"threads_started":1}
W20260806 11:42:00.392138 452 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5cc)
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:00.393206 452 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:00.394145 452 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:42:00.403959 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:00.395033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.0.113.1:36191)
0806 11:42:00.395177 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:00.395181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:00.395305 (+ 124us) server_negotiation.cc:407] Connection header received
0806 11:42:00.395437 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:00.395440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:00.395482 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:00.395550 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:00.396031 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.396850 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.397651 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.398012 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.398094 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:00.400526 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:00.400535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:00.400547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:00.400570 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:00.402112 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.402560 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.402565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.402566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.402645 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.402939 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.402943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.402944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.403192 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:00.403398 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:00.403607 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 11:42:00.403801 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 11:42:00.407769 9144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:00.406705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.0.113.1:36191)
0806 11:42:00.406839 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:00.406843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:00.406960 (+ 117us) server_negotiation.cc:407] Connection header received
0806 11:42:00.407074 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:00.407077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:00.407119 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:00.407202 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:00.407685 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43282: BlockingRecv error: recv got EOF from 127.0.0.1:43282 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 11:42:00.408048 452 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5cc' of type FILE
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:00.409049 452 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:00.410006 452 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:42:00.417948 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:00.410801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.0.113.1:36191)
0806 11:42:00.410961 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:00.410964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:00.410989 (+ 25us) server_negotiation.cc:407] Connection header received
0806 11:42:00.411092 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:00.411094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:00.411125 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:00.411185 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:00.411578 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.412066 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.412826 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.413044 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.413118 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:00.415134 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:00.415137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:00.415138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:00.415158 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:00.416557 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.416920 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.416922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.416923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.416990 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.417202 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.417204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.417205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.417341 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:00.417431 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:00.417661 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 11:42:00.417786 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 11:42:00.420231 9144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:00.419447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.0.113.1:36191)
0806 11:42:00.419526 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:00.419529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:00.419694 (+ 165us) server_negotiation.cc:407] Connection header received
0806 11:42:00.419755 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:00.419757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:00.419788 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:00.419845 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:00.420148 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43300: BlockingRecv error: recv got EOF from 127.0.0.1:43300 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260806 11:42:00.420365 452 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786016488822243-452-0/krb5kdc/krb5cc' of type FILE
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:00.421254 452 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:00.422151 452 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:42:00.429498 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:00.422870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.0.113.1:36191)
0806 11:42:00.423001 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:00.423004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:00.423016 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:42:00.423129 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:00.423131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:00.423160 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:00.423219 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:00.423597 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.424041 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.424823 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:00.425019 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:00.425092 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:00.426981 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:00.426983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:00.426985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:00.427004 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:00.428164 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.428574 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.428576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.428577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.428648 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:00.428856 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:00.428858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:00.428859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:00.428991 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:00.429074 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:00.429227 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 11:42:00.429301 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:42:00.430380 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 8555
I20260806 11:42:00.444896 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 8746
I20260806 11:42:00.454396 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 8934
I20260806 11:42:00.470505 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 8414
2026-08-06T11:42:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2418 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[9159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[9159](info): set up 2 sockets
Aug 06 11:42:00 dist-test-slave-45dx krb5kdc[9159](info): commencing operation
krb5kdc: starting...
W20260806 11:42:01.680656 452 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:42:01 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:42:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.62.keytab.
Entry for principal oryx/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.62.keytab.
I20260806 11:42:01.807030 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:44983
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.62.keytab
--principal=oryx/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:44983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:01.916138 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:01.916471 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:01.916551 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:01.920449 9174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:01.920526 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:01.920563 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:01.920599 9174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:01.920632 9174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:01.925774 9174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:44983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.62.keytab
--principal=oryx/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:44983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:01.926752 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:01.927747 9174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:01.936259 9181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:01.936266 9180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:01.936275 9184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:01.936411 9174 server_base.cc:1034] running on GCE node
I20260806 11:42:01.937614 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:01.938707 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:01.939924 9174 hybrid_clock.cc:648] HybridClock initialized: now 1786016521939903 us; error 32 us; skew 500 ppm
Aug 06 11:42:01 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016521, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:01.942847 9174 init.cc:377] Logged in from keytab as oryx/127.0.113.62@KRBTEST.COM (short username oryx)
I20260806 11:42:01.943691 9174 webserver.cc:460] Webserver started at http://127.0.113.62:37687/ using document root <none> and password file <none>
I20260806 11:42:01.944218 9174 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:01.944276 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:01.944521 9174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:01.946579 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "2fac143e703c4cb19d4b5ca844af0183"
format_stamp: "Formatted at 2026-08-06 11:42:01 on dist-test-slave-45dx"
server_key: "7373191d56478380c0b80da1e56140e8"
server_key_iv: "1842f41ace0d697bfe3fbd77aa912d82"
server_key_version: "encryptionkey@0"
I20260806 11:42:01.947117 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "2fac143e703c4cb19d4b5ca844af0183"
format_stamp: "Formatted at 2026-08-06 11:42:01 on dist-test-slave-45dx"
server_key: "7373191d56478380c0b80da1e56140e8"
server_key_iv: "1842f41ace0d697bfe3fbd77aa912d82"
server_key_version: "encryptionkey@0"
I20260806 11:42:01.950803 9174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:42:01.953208 9199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:01.954421 9174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260806 11:42:01.954550 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "2fac143e703c4cb19d4b5ca844af0183"
format_stamp: "Formatted at 2026-08-06 11:42:01 on dist-test-slave-45dx"
server_key: "7373191d56478380c0b80da1e56140e8"
server_key_iv: "1842f41ace0d697bfe3fbd77aa912d82"
server_key_version: "encryptionkey@0"
I20260806 11:42:01.954653 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:01.965937 9174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:01.968819 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:01.969036 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:01.976418 9174 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:44983
I20260806 11:42:01.976688 9282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:44983 every 8 connection(s)
I20260806 11:42:01.977416 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:42:01.980321 9283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:01.983059 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 9174
I20260806 11:42:01.983174 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:42:01.983500 452 external_mini_cluster.cc:1428] Setting key 595933377c6da9aaea92278bcf4b6ac2
I20260806 11:42:01.986451 9283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: Bootstrap starting.
I20260806 11:42:01.989108 9283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:01.989885 9283 log.cc:824] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:01.991969 9283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: No bootstrap required, opened a new log
Aug 06 11:42:01 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:01.996097 9283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } }
I20260806 11:42:01.996398 9283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:01.996479 9283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fac143e703c4cb19d4b5ca844af0183, State: Initialized, Role: FOLLOWER
I20260806 11:42:01.996943 9283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } }
I20260806 11:42:01.997105 9283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:42:01.997188 9283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:42:01.997316 9283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:01.998488 9283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } }
I20260806 11:42:01.998838 9283 leader_election.cc:302] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fac143e703c4cb19d4b5ca844af0183; no voters:
I20260806 11:42:01.999181 9283 leader_election.cc:288] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:42:01.999298 9289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:01.999544 9289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 1 LEADER]: Becoming Leader. State: Replica: 2fac143e703c4cb19d4b5ca844af0183, State: Running, Role: LEADER
I20260806 11:42:01.999641 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:01.984937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.0.113.62:44983)
0806 11:42:01.985363 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:01.985384 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:01.985429 (+ 45us) server_negotiation.cc:407] Connection header received
0806 11:42:01.986238 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:01.986260 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:01.986665 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:01.987019 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:01.987725 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:01.989202 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:01.990110 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:01.990683 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:01.990841 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:01.993589 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:01.993595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:01.993635 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:01.993666 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:01.996295 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:01.996881 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:01.996887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:01.996892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:01.997048 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:01.997418 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:01.997423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:01.997424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:01.997936 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:01.998145 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:01.998376 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 11:42:01.998678 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260806 11:42:01.999895 9289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } }
I20260806 11:42:02.000450 9283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:42:02.001931 9291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } }
I20260806 11:42:02.002071 9291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:02.002483 9293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fac143e703c4cb19d4b5ca844af0183. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } }
I20260806 11:42:02.002620 9293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
W20260806 11:42:02.005287 9309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:42:02.005380 9309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:42:02.005487 9306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:42:02.006255 9306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:42:02.011799 9306 catalog_manager.cc:1347] Generated new cluster ID: 977f3e83e3b54ec4b71ace2eccabe00f
I20260806 11:42:02.011861 9306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:42:02.024489 9306 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:42:02.025760 9306 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:42:02.043468 9306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: Generated new TSK 0
I20260806 11:42:02.044380 9306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab.
Entry for principal oryx/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab.
I20260806 11:42:02.079180 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:44983
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab
--principal=oryx/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:02.192668 9317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:02.192931 9317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:02.192974 9317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:02.197167 9317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:02.197221 9317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:02.197278 9317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:42:02.202579 9317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab
--principal=oryx/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:44983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:02.203590 9317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:02.204698 9317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:02.214468 9329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:02.214478 9326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:02.214478 9324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:02.214514 9317 server_base.cc:1034] running on GCE node
I20260806 11:42:02.215191 9317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:02.215790 9317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:02.216984 9317 hybrid_clock.cc:648] HybridClock initialized: now 1786016522216973 us; error 30 us; skew 500 ppm
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016522, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:02.219700 9317 init.cc:377] Logged in from keytab as oryx/127.0.113.1@KRBTEST.COM (short username oryx)
I20260806 11:42:02.220575 9317 webserver.cc:460] Webserver started at http://127.0.113.1:43013/ using document root <none> and password file <none>
I20260806 11:42:02.221138 9317 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:02.221230 9317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:02.221450 9317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:02.223415 9317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "dd8f1735feb8499f95df54292701beec"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "5f9068f36df5a16fcf4c6f8661f43993"
server_key_iv: "3316fd0ccfe1851343e56d624bed227d"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.223965 9317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "dd8f1735feb8499f95df54292701beec"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "5f9068f36df5a16fcf4c6f8661f43993"
server_key_iv: "3316fd0ccfe1851343e56d624bed227d"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.227437 9317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:42:02.229794 9342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:02.231001 9317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:02.231128 9317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "dd8f1735feb8499f95df54292701beec"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "5f9068f36df5a16fcf4c6f8661f43993"
server_key_iv: "3316fd0ccfe1851343e56d624bed227d"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.231230 9317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:02.248075 9317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:02.250855 9317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:02.251067 9317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:02.251613 9317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:02.252449 9317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:02.252516 9317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:02.252580 9317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:02.252625 9317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:02.261611 9317 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:33483
I20260806 11:42:02.261638 9509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:33483 every 8 connection(s)
I20260806 11:42:02.262552 9317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:42:02.265782 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 9317
I20260806 11:42:02.265908 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:42:02.266351 452 external_mini_cluster.cc:1428] Setting key 75ba42d947df8b45e56645ac4bde13b9
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016522, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:02.279479 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.264943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:60233 (local address 127.0.113.62:44983)
0806 11:42:02.265118 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:02.265123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:02.266504 (+ 1381us) server_negotiation.cc:407] Connection header received
0806 11:42:02.268244 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:02.268248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:02.268327 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:02.268420 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:02.270022 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.270957 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.272201 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.272484 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.272569 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.275297 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:02.275302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:02.275304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:02.275336 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:02.277302 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.277814 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.277819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.277821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.277928 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.278273 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.278277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.278280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.278528 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:02.278633 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:02.279124 (+ 491us) server_negotiation.cc:299] Negotiation successful
0806 11:42:02.279269 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for oryx/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:42:02.280166 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.265403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.1:60233)
0806 11:42:02.266362 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:02.266412 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:02.268033 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:02.268596 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:02.268606 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:02.269141 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:02.269850 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.269872 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.271083 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.271088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.272083 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.272093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.272642 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.272648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.272872 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.273516 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:02.273543 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:02.275124 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:02.277426 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.277431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.277442 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.277701 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.278036 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.278038 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.278039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.278173 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.278639 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:02.278643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:02.278868 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 11:42:02.279031 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 11:42:02.279268 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":173,"threads_started":1}
I20260806 11:42:02.281570 9510 heartbeater.cc:347] Connected to a master server at 127.0.113.62:44983
I20260806 11:42:02.281834 9510 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:02.282384 9510 heartbeater.cc:511] Master 127.0.113.62:44983 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.2@KRBTEST.COM" created.
I20260806 11:42:02.283987 9217 ts_manager.cc:194] Registered new tserver with Master: dd8f1735feb8499f95df54292701beec (127.0.113.1:33483)
I20260806 11:42:02.285192 9217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:60233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab.
Entry for principal oryx/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab.
I20260806 11:42:02.299379 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:44983
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab
--principal=oryx/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:02.412153 9519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:02.412478 9519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:02.412525 9519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:02.416280 9519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:02.416335 9519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:02.416400 9519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:42:02.421222 9519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab
--principal=oryx/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:44983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:02.422101 9519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:02.422994 9519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:02.432049 9528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:02.432117 9530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:02.432253 9519 server_base.cc:1034] running on GCE node
W20260806 11:42:02.432459 9532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:02.432888 9519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:02.433475 9519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:02.434677 9519 hybrid_clock.cc:648] HybridClock initialized: now 1786016522434666 us; error 45 us; skew 500 ppm
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016522, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:02.437703 9519 init.cc:377] Logged in from keytab as oryx/127.0.113.2@KRBTEST.COM (short username oryx)
I20260806 11:42:02.438529 9519 webserver.cc:460] Webserver started at http://127.0.113.2:44295/ using document root <none> and password file <none>
I20260806 11:42:02.439079 9519 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:02.439167 9519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:02.439397 9519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:02.441396 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "6850047bf5d34a4fa628dd2366429165"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "e2f7d0e920d573ea88ae01fd428aca70"
server_key_iv: "bb5270683acd858b04b6aad9ec2d5c4a"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.441991 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "6850047bf5d34a4fa628dd2366429165"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "e2f7d0e920d573ea88ae01fd428aca70"
server_key_iv: "bb5270683acd858b04b6aad9ec2d5c4a"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.445564 9519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:42:02.447881 9544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:02.449199 9519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:02.449333 9519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "6850047bf5d34a4fa628dd2366429165"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "e2f7d0e920d573ea88ae01fd428aca70"
server_key_iv: "bb5270683acd858b04b6aad9ec2d5c4a"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.449441 9519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:02.475358 9519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:02.478703 9519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:02.478917 9519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:02.479534 9519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:02.480402 9519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:02.480473 9519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:02.480547 9519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:02.480590 9519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:02.489537 9519 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:33957
I20260806 11:42:02.489565 9713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:33957 every 8 connection(s)
I20260806 11:42:02.490417 9519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:42:02.496366 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 9519
I20260806 11:42:02.496495 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:42:02.496796 452 external_mini_cluster.cc:1428] Setting key c8ddfac30aff59c0a2842bd768a0e05a
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016522, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:02.505110 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.492203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:46673 (local address 127.0.113.62:44983)
0806 11:42:02.492405 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:02.492411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:02.493718 (+ 1307us) server_negotiation.cc:407] Connection header received
0806 11:42:02.494583 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:02.494586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:02.494629 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:02.494701 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:02.495826 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.496548 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.497768 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.498022 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.498139 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.500850 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:02.500856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:02.500861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:02.500893 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:02.502782 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.503324 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.503331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.503336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.503460 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.503900 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.503904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.503906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.504116 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:02.504248 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:02.504775 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 11:42:02.504902 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:42:02.505931 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.492643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.2:46673)
0806 11:42:02.493558 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:02.493600 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:02.494401 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:02.494887 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:02.494893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:02.495246 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:02.495695 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.495707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.496699 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.496702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.497622 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.497633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.498181 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.498186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.498359 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.498989 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:02.499005 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:02.500639 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:02.502912 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.502921 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.502937 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.503212 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.503619 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.503622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.503623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.503758 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.504256 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:02.504264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:02.504535 (+ 271us) client_negotiation.cc:769] Sending connection context
0806 11:42:02.504720 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 11:42:02.505113 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":537,"threads_started":1}
I20260806 11:42:02.507203 9714 heartbeater.cc:347] Connected to a master server at 127.0.113.62:44983
I20260806 11:42:02.507444 9714 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:02.507927 9714 heartbeater.cc:511] Master 127.0.113.62:44983 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:42:02.509037 9217 ts_manager.cc:194] Registered new tserver with Master: 6850047bf5d34a4fa628dd2366429165 (127.0.113.2:33957)
I20260806 11:42:02.509693 9217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:46673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab.
Entry for principal oryx/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab.
I20260806 11:42:02.527581 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:44983
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab
--principal=oryx/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:02.641556 9720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:02.641826 9720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:02.641973 9720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:02.646003 9720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:02.646057 9720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:02.646114 9720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:42:02.651257 9720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab
--principal=oryx/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:44983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:02.652412 9720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:02.653535 9720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:02.663913 9733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:02.664084 9720 server_base.cc:1034] running on GCE node
W20260806 11:42:02.664029 9728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:02.663913 9727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:02.664669 9720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:02.665331 9720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:02.666522 9720 hybrid_clock.cc:648] HybridClock initialized: now 1786016522666510 us; error 29 us; skew 500 ppm
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016522, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:02.669512 9720 init.cc:377] Logged in from keytab as oryx/127.0.113.3@KRBTEST.COM (short username oryx)
I20260806 11:42:02.670468 9720 webserver.cc:460] Webserver started at http://127.0.113.3:36805/ using document root <none> and password file <none>
I20260806 11:42:02.671056 9720 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:02.671154 9720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:02.671399 9720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:02.673558 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "11654e1660a64ad7a09ed46610715c1b"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "dccdb16ae2dc8487e345fbc78637c9c8"
server_key_iv: "391c20447a7710dd73c0abfdec3a9cfa"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.674237 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "11654e1660a64ad7a09ed46610715c1b"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "dccdb16ae2dc8487e345fbc78637c9c8"
server_key_iv: "391c20447a7710dd73c0abfdec3a9cfa"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.678154 9720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 11:42:02.680696 9743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:02.682087 9720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:42:02.682233 9720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "11654e1660a64ad7a09ed46610715c1b"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "dccdb16ae2dc8487e345fbc78637c9c8"
server_key_iv: "391c20447a7710dd73c0abfdec3a9cfa"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.682353 9720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:02.707566 9720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:02.710453 9720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:02.710665 9720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:02.711268 9720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:02.712126 9720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:02.712198 9720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:02.712272 9720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:02.712338 9720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 11:42:02.721946 9720 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:43973
I20260806 11:42:02.721977 9897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:43973 every 8 connection(s)
I20260806 11:42:02.722818 9720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:42:02.724757 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 9720
I20260806 11:42:02.724886 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:42:02.725227 452 external_mini_cluster.cc:1428] Setting key f6e79b40c8f6aeadc96fd1edac1de3e2
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016522, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:02.736781 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.724563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:33043 (local address 127.0.113.62:44983)
0806 11:42:02.724703 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:02.724707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:02.726337 (+ 1630us) server_negotiation.cc:407] Connection header received
0806 11:42:02.727499 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:02.727503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:02.727554 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:02.727629 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:02.728814 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.729327 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.730496 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.730696 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.730770 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.733300 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:02.733303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:02.733305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:02.733327 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:02.734841 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.735336 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.735339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.735341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.735438 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.735788 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.735791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.735792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.735986 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:02.736063 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:02.736568 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 11:42:02.736666 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:42:02.737550 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.725016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.3:33043)
0806 11:42:02.726148 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:02.726212 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:02.727268 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:02.727823 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:02.727830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:02.728169 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:02.728692 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.728706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.729481 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.729488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.730375 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.730385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.730891 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.730895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.731046 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.731661 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:02.731677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:02.733153 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:02.734946 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.734951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.734961 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.735211 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.735509 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.735511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.735512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.735704 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.736077 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:02.736082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:02.736290 (+ 208us) client_negotiation.cc:769] Sending connection context
0806 11:42:02.736519 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 11:42:02.736752 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":743,"threads_started":1}
I20260806 11:42:02.738852 9898 heartbeater.cc:347] Connected to a master server at 127.0.113.62:44983
I20260806 11:42:02.739113 9898 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:02.739646 9898 heartbeater.cc:511] Master 127.0.113.62:44983 requested a full tablet report, sending...
I20260806 11:42:02.740595 9215 ts_manager.cc:194] Registered new tserver with Master: 11654e1660a64ad7a09ed46610715c1b (127.0.113.3:43973)
I20260806 11:42:02.741135 9215 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:33043
I20260806 11:42:02.751049 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.61.keytab.
Entry for principal oryx/127.0.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.61.keytab.
I20260806 11:42:02.779528 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.61:39161
--webserver_interface=127.0.113.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.61.keytab
--principal=oryx/127.0.113.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:44983,127.0.113.61:39161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:02.890798 9907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:02.891041 9907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:02.891078 9907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:02.895030 9907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:02.895128 9907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:02.895165 9907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:02.895182 9907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:02.895197 9907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:02.900192 9907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:44983,127.0.113.61:39161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.61.keytab
--principal=oryx/127.0.113.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.61:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.113.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
W20260806 11:42:02.900429 9907 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.113.62:44983,127.0.113.61:39161'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 11:42:02.901266 9907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:02.902498 9907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:02.910926 9920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:02.910917 9917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:02.911104 9918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:02.911336 9907 server_base.cc:1034] running on GCE node
I20260806 11:42:02.911904 9907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:02.913074 9907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:02.914254 9907 hybrid_clock.cc:648] HybridClock initialized: now 1786016522914226 us; error 39 us; skew 500 ppm
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016522, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:02.917281 9907 init.cc:377] Logged in from keytab as oryx/127.0.113.61@KRBTEST.COM (short username oryx)
I20260806 11:42:02.918092 9907 webserver.cc:460] Webserver started at http://127.0.113.61:43707/ using document root <none> and password file <none>
I20260806 11:42:02.918578 9907 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:02.918635 9907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:02.918794 9907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:02.920822 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data/instance:
uuid: "f78d76dabc0946e7a419656c7bac1a81"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "7f4e70d7e4d6cdcef986019af8341e71"
server_key_iv: "f453f7d5cb18b641bbb7ed16c41382fe"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.921339 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal/instance:
uuid: "f78d76dabc0946e7a419656c7bac1a81"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "7f4e70d7e4d6cdcef986019af8341e71"
server_key_iv: "f453f7d5cb18b641bbb7ed16c41382fe"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.925079 9907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 11:42:02.927345 9928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:02.928581 9907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:02.928718 9907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal
uuid: "f78d76dabc0946e7a419656c7bac1a81"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "7f4e70d7e4d6cdcef986019af8341e71"
server_key_iv: "f453f7d5cb18b641bbb7ed16c41382fe"
server_key_version: "encryptionkey@0"
I20260806 11:42:02.928833 9907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:02.940958 9907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:02.943591 9907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:02.943783 9907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:02.950667 9907 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.61:39161
I20260806 11:42:02.950940 10000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.61:39161 every 8 connection(s)
I20260806 11:42:02.951623 9907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
I20260806 11:42:02.954481 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:02.956130 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 9907
I20260806 11:42:02.958734 10001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.113.61@KRBTEST.COM
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016522, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.61@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:02.969844 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.960896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.0.113.62:44983)
0806 11:42:02.961045 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:02.961050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:02.961417 (+ 367us) server_negotiation.cc:407] Connection header received
0806 11:42:02.961537 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:02.961539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:02.961588 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:02.961670 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:02.962314 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.962830 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.963786 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.964230 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.964382 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.966529 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:02.966532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:02.966534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:02.966558 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:02.967877 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.968417 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.968420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.968424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.968506 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.968845 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.968848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.968850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.969069 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:02.969180 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:02.969546 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 11:42:02.969667 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 11:42:02.970639 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.957761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.0.113.61:39161)
0806 11:42:02.958207 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:02.958226 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:02.958269 (+ 43us) server_negotiation.cc:407] Connection header received
0806 11:42:02.959067 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:02.959090 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:02.959499 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:02.959864 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:02.960604 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.961862 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.962688 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.963148 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.963322 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.966112 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:02.966118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:02.966127 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:02.966151 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:02.967928 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.968438 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.968444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.968446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.968550 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.968857 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.968860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.968862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.969210 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:02.969435 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:02.969661 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 11:42:02.970066 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":201,"server-negotiator.queue_time_us":295,"thread_start_us":109,"threads_started":1}
I20260806 11:42:02.970638 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.960810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:59670)
0806 11:42:02.961258 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:02.961303 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:02.961438 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:02.961722 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:02.961726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:02.962063 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:02.962183 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.962188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.962979 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.962982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.963663 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.963674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.964422 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.964425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.964512 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.965036 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:02.965052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:02.966368 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:02.968002 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.968010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.968019 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.968254 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.968611 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.968614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.968616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.968739 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.969178 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:02.969185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:02.969322 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 11:42:02.969487 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 11:42:02.969811 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":124,"threads_started":1}
I20260806 11:42:02.972193 10001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 11:42:02 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016522, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.61@KRBTEST.COM for oryx/127.0.113.61@KRBTEST.COM
I20260806 11:42:02.980916 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.972983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.0.113.61:39161)
0806 11:42:02.973072 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:02.973076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:02.973125 (+ 49us) server_negotiation.cc:407] Connection header received
0806 11:42:02.973237 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:02.973239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:02.973277 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:02.973357 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:02.973723 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.974186 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.974876 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.975094 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.975197 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.977860 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:02.977863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:02.977865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:02.977893 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:02.979326 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.979746 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.979751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.979753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.979830 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:02.980095 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:02.980100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:02.980102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:02.980256 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:02.980410 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:02.980671 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 11:42:02.980798 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":26}
I20260806 11:42:02.980916 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:02.972910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.0.1:52378)
0806 11:42:02.973070 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:02.973084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:02.973161 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:02.973367 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:02.973370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:02.973543 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:02.973615 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.973620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.974285 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.974291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.974766 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:02.974773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:02.975195 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:02.975199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:02.975308 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:02.975756 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:02.975770 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:02.977733 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:02.979458 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.979461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.979463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.979651 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.979919 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:02.979924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:02.979925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:02.980008 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:02.980414 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:02.980418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:02.980511 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 11:42:02.980614 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 11:42:02.980746 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":18}
I20260806 11:42:02.983952 10001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: Bootstrap starting.
I20260806 11:42:02.986591 10001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:02.987406 10001 log.cc:824] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:02.989445 10001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: No bootstrap required, opened a new log
I20260806 11:42:02.992486 10001 raft_consensus.cc:348] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } }
I20260806 11:42:02.992662 10001 raft_consensus.cc:374] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:02.992722 10001 raft_consensus.cc:729] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78d76dabc0946e7a419656c7bac1a81, State: Initialized, Role: FOLLOWER
I20260806 11:42:02.993114 10001 consensus_queue.cc:260] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } }
I20260806 11:42:02.993522 10009 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } } }
I20260806 11:42:02.993659 10009 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:42:02.994185 10001 sys_catalog.cc:564] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 11:42:02.998912 10026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:42:02.998999 10026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:42:03.002097 9907 master_runner.cc:428] Detected that this master f78d76dabc0946e7a419656c7bac1a81 is joining an existing cluster
I20260806 11:42:03.002161 9907 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.113.61:39161
I20260806 11:42:03.014045 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.004479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.0.1:52388)
0806 11:42:03.005120 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.005135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.005263 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.005686 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.005689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.006083 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.006342 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.006350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.007400 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.007403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.009079 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.009088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.009685 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.009689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.009848 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.010329 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.010344 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.010674 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.012462 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.012467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.012470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.012783 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.013083 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.013089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.013093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.013191 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.013616 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.013619 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.013711 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.013799 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.013902 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":123,"spinlock_wait_cycles":13952,"thread_start_us":117,"threads_started":1}
I20260806 11:42:03.014175 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.004623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.0.113.61:39161)
0806 11:42:03.004729 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.004733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.005230 (+ 497us) server_negotiation.cc:407] Connection header received
0806 11:42:03.005375 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.005378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.005426 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.005519 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.006465 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.007255 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.009232 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.009575 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.009679 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.010799 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.010805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.010809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.010835 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.012347 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.012887 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.012890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.012892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.012980 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.013292 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.013295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.013296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.013505 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.013602 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.013874 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.013991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260806 11:42:03.014810 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.004838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:59686)
0806 11:42:03.005143 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.005158 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.005290 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.005693 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.005696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.006079 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.006347 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.006352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.007400 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.007403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.008449 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.008476 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.009064 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.009067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.009172 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.009768 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.009782 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.010318 (+ 536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.012406 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.012410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.012415 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.012783 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.013133 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.013139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.013142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.013212 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.014346 (+ 1134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.014349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.014432 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.014522 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.014649 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":113,"spinlock_wait_cycles":3072,"thread_start_us":127,"threads_started":1}
I20260806 11:42:03.015489 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.004497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.0.113.62:44983)
0806 11:42:03.004646 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.004651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.005221 (+ 570us) server_negotiation.cc:407] Connection header received
0806 11:42:03.005420 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.005423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.005475 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.005570 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.006479 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.007255 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.008603 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.008934 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.009045 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.010462 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.010466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.010469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.010496 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.012277 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.012915 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.012919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.012920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.013009 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.013329 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.013334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.013336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.013515 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.013631 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.015174 (+ 1543us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.015322 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016521, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.62@KRBTEST.COM for oryx/127.0.113.61@KRBTEST.COM
I20260806 11:42:03.028568 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.019605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.0.113.61:39161)
0806 11:42:03.019706 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.019710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.020056 (+ 346us) server_negotiation.cc:407] Connection header received
0806 11:42:03.020256 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.020259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.020359 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.020437 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.021068 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.021537 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.022400 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.022617 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.022717 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.025341 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.025347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.025352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.025382 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.026852 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.027300 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.027305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.027306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.027383 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.027663 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.027666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.027667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.027833 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.027960 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.028281 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.028426 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 11:42:03.028604 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.019558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.0.1:52398)
0806 11:42:03.019936 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.019977 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.020123 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.020480 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.020486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.020855 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.020953 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.020958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.021644 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.021648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.022278 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.022287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.022730 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.022734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.022855 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.023374 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.023402 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.025185 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.026961 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.026972 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.026978 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.027201 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.027482 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.027485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.027487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.027570 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.027964 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.027973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.028086 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.028227 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.028447 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":76,"threads_started":1}
I20260806 11:42:03.030222 9215 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.113.61:39161: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81"
member_type: NON_VOTER
last_known_addr {
host: "127.0.113.61"
port: 39161
}
attrs {
promote: true
}
}
dest_uuid: "2fac143e703c4cb19d4b5ca844af0183"
cas_config_opid_index: -1
I20260806 11:42:03.030941 9215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } }
I20260806 11:42:03.033146 10041 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f78d76dabc0946e7a419656c7bac1a81 (127.0.113.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } } }
I20260806 11:42:03.033663 9966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:03.035127 9966 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Refusing update from remote peer 2fac143e703c4cb19d4b5ca844af0183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 11:42:03.035290 10041 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.113.61:39161
WARNING: no policy specified for oryx/127.0.113.60@KRBTEST.COM; defaulting to no policy
I20260806 11:42:03.035737 10041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } } }
I20260806 11:42:03.035837 9907 master.cc:537] Master@127.0.113.61:39161 shutting down...
I20260806 11:42:03.036084 10043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:03.035959 10041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.60@KRBTEST.COM" created.
I20260806 11:42:03.040545 9907 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:03.040853 9907 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:03.041003 9907 tablet_replica.cc:330] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: stopping tablet replica
W20260806 11:42:03.041045 9201 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 11:42:03.042049 9201 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 -> Peer f78d76dabc0946e7a419656c7bac1a81 (127.0.113.61:39161): Couldn't send request to peer f78d76dabc0946e7a419656c7bac1a81. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 11:42:03.056515 9907 master.cc:552] Master@127.0.113.61:39161 shutdown complete.
I20260806 11:42:03.056589 9907 master_runner.cc:305] Clearing existing system tablet
I20260806 11:42:03.056857 9907 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:03.056908 9907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:03.058885 9907 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:03.059854 10049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.060118 9907 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:03.060252 9907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:03.060288 9907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal
uuid: "f78d76dabc0946e7a419656c7bac1a81"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "7f4e70d7e4d6cdcef986019af8341e71"
server_key_iv: "f453f7d5cb18b641bbb7ed16c41382fe"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.60.keytab.
Entry for principal oryx/127.0.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.60.keytab.
I20260806 11:42:03.062567 9907 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:03.062821 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.60:46597
--webserver_interface=127.0.113.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.60.keytab
--principal=oryx/127.0.113.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:44983,127.0.113.61:39161,127.0.113.60:46597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:42:03.064910 9907 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 11:42:03.065003 9907 log.cc:1196] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 11:42:03.065276 9907 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: Deleting consensus metadata
I20260806 11:42:03.065452 9907 master_runner.cc:315] Copying system tablet from 127.0.113.62:44983
I20260806 11:42:03.066795 9907 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: tablet copy: Beginning tablet copy session from remote peer at address 127.0.113.62:44983
I20260806 11:42:03.077310 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.067571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.0.113.62:44983)
0806 11:42:03.067697 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.067702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.067982 (+ 280us) server_negotiation.cc:407] Connection header received
0806 11:42:03.068137 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.068140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.068192 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.068290 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.068987 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.069808 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.071197 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.071551 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.071657 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.073237 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.073241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.073244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.073272 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.075259 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.075868 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.075872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.075874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.075968 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.076296 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.076344 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.076346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.076596 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.076712 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.077017 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.077138 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:42:03.077319 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.067490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:59688)
0806 11:42:03.067860 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.067878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.068028 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.068394 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.068400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.068650 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.068842 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.068855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.069949 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.069955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.071057 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.071073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.071695 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.071700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.071831 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.072522 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.072545 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.073068 (+ 523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.075408 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.075415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.075419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.075736 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.076080 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.076084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.076086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.076194 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.076717 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.076720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.076833 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.076943 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.077100 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":93,"threads_started":1}
I20260806 11:42:03.078289 9253 tablet_copy_service.cc:140] P 2fac143e703c4cb19d4b5ca844af0183: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f78d76dabc0946e7a419656c7bac1a81 ({username='oryx', principal='oryx/127.0.113.61@KRBTEST.COM'} at 127.0.0.1:59688)
I20260806 11:42:03.078488 9253 tablet_copy_service.cc:161] P 2fac143e703c4cb19d4b5ca844af0183: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f78d76dabc0946e7a419656c7bac1a81 at {username='oryx', principal='oryx/127.0.113.61@KRBTEST.COM'} at 127.0.0.1:59688: session id = f78d76dabc0946e7a419656c7bac1a81-00000000000000000000000000000000
I20260806 11:42:03.080461 9253 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 11:42:03.082355 9907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 11:42:03.084884 9907 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: tablet copy: Starting download of 0 data blocks...
I20260806 11:42:03.085089 9907 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: tablet copy: Starting download of 1 WAL segments...
I20260806 11:42:03.087087 9907 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 11:42:03.088851 9253 tablet_copy_service.cc:342] P 2fac143e703c4cb19d4b5ca844af0183: Request end of tablet copy session f78d76dabc0946e7a419656c7bac1a81-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.113.61@KRBTEST.COM'} at 127.0.0.1:59688
I20260806 11:42:03.088972 9253 tablet_copy_service.cc:433] P 2fac143e703c4cb19d4b5ca844af0183: ending tablet copy session f78d76dabc0946e7a419656c7bac1a81-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f78d76dabc0946e7a419656c7bac1a81
I20260806 11:42:03.096884 9907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:03.098511 10062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.098703 10063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.098889 10065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:03.100371 9907 server_base.cc:1034] running on GCE node
I20260806 11:42:03.100526 9907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:03.100893 9907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:03.102048 9907 hybrid_clock.cc:648] HybridClock initialized: now 1786016523102002 us; error 63 us; skew 500 ppm
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:03.103824 9907 init.cc:377] Logged in from keytab as oryx/127.0.113.61@KRBTEST.COM (short username oryx)
I20260806 11:42:03.104346 9907 webserver.cc:460] Webserver started at http://127.0.113.61:43229/ using document root <none> and password file <none>
I20260806 11:42:03.104611 9907 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:03.104699 9907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:03.106037 9907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:03.108492 10075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.108743 9907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:03.108830 9907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal
uuid: "f78d76dabc0946e7a419656c7bac1a81"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "7f4e70d7e4d6cdcef986019af8341e71"
server_key_iv: "f453f7d5cb18b641bbb7ed16c41382fe"
server_key_version: "encryptionkey@0"
I20260806 11:42:03.108914 9907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:03.115760 9907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:03.118104 9907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:03.122874 9907 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.61:39161
I20260806 11:42:03.123023 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.61:39161 every 8 connection(s)
I20260806 11:42:03.123461 9907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
I20260806 11:42:03.125474 10162 sys_catalog.cc:263] Verifying existing consensus state
I20260806 11:42:03.126574 10162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: Bootstrap starting.
I20260806 11:42:03.137941 10162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:42:03.138309 10162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: Bootstrap complete.
I20260806 11:42:03.138967 10162 raft_consensus.cc:348] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } }
I20260806 11:42:03.139128 10162 raft_consensus.cc:729] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f78d76dabc0946e7a419656c7bac1a81, State: Initialized, Role: LEARNER
I20260806 11:42:03.139252 10162 consensus_queue.cc:260] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } }
I20260806 11:42:03.139657 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } } }
I20260806 11:42:03.139775 10162 sys_catalog.cc:564] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:42:03.139859 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: This master's current role is: LEARNER
I20260806 11:42:03.143854 10186 catalog_manager.cc:1259] Loaded cluster ID: 977f3e83e3b54ec4b71ace2eccabe00f
I20260806 11:42:03.144006 10186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: loading cluster ID for follower catalog manager: success
I20260806 11:42:03.146171 10186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: acquiring CA information for follower catalog manager: success
I20260806 11:42:03.147624 10186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 11:42:03.195966 10051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:03.196220 10051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:03.196259 10051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:03.200374 10051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:03.200428 10051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:03.200449 10051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:03.200492 10051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:03.200521 10051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:03.205941 10051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:44983,127.0.113.61:39161,127.0.113.60:46597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.60.keytab
--principal=oryx/127.0.113.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.60:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.113.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:03.206996 10051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:03.208000 10051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:03.216337 10192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.216339 10195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.216338 10191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:03.216594 10051 server_base.cc:1034] running on GCE node
I20260806 11:42:03.217147 10051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:03.218149 10051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:03.219358 10051 hybrid_clock.cc:648] HybridClock initialized: now 1786016523219330 us; error 26 us; skew 500 ppm
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:03.222389 10051 init.cc:377] Logged in from keytab as oryx/127.0.113.60@KRBTEST.COM (short username oryx)
I20260806 11:42:03.223163 10051 webserver.cc:460] Webserver started at http://127.0.113.60:41101/ using document root <none> and password file <none>
I20260806 11:42:03.223655 10051 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:03.223747 10051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:03.223968 10051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:03.225953 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data/instance:
uuid: "55b6cc576f2a45cfbcd7a26633595361"
format_stamp: "Formatted at 2026-08-06 11:42:03 on dist-test-slave-45dx"
server_key: "7e6f197e65713c383ed6afb3b90c226a"
server_key_iv: "41cc06d9292fa4e2f5ab47c31a1f5b4b"
server_key_version: "encryptionkey@0"
I20260806 11:42:03.226485 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal/instance:
uuid: "55b6cc576f2a45cfbcd7a26633595361"
format_stamp: "Formatted at 2026-08-06 11:42:03 on dist-test-slave-45dx"
server_key: "7e6f197e65713c383ed6afb3b90c226a"
server_key_iv: "41cc06d9292fa4e2f5ab47c31a1f5b4b"
server_key_version: "encryptionkey@0"
I20260806 11:42:03.230177 10051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 11:42:03.232514 10205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.233695 10051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:03.233789 10051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal
uuid: "55b6cc576f2a45cfbcd7a26633595361"
format_stamp: "Formatted at 2026-08-06 11:42:03 on dist-test-slave-45dx"
server_key: "7e6f197e65713c383ed6afb3b90c226a"
server_key_iv: "41cc06d9292fa4e2f5ab47c31a1f5b4b"
server_key_version: "encryptionkey@0"
I20260806 11:42:03.233929 10051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:03.251032 10051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:03.254282 10051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:03.254437 10051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:03.261678 10051 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.60:46597
I20260806 11:42:03.262025 10282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.60:46597 every 8 connection(s)
I20260806 11:42:03.262636 10051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
I20260806 11:42:03.266003 10283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:03.269908 10283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:42:03.270639 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 10051
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.60@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.113.60@KRBTEST.COM
I20260806 11:42:03.283389 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.271813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.0.113.62:44983)
0806 11:42:03.271952 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.271956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.272680 (+ 724us) server_negotiation.cc:407] Connection header received
0806 11:42:03.273535 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.273539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.273616 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.273739 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.274660 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.275178 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.276281 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.276521 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.276595 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.279305 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.279314 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.279316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.279341 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.281018 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.281686 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.281689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.281690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.281762 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.282185 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.282191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.282194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.282384 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.282499 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.283090 (+ 591us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.283216 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:42:03.284135 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.272096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:59694)
0806 11:42:03.272550 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.272584 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.273306 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.273811 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.273816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.274216 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.274509 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.274523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.275316 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.275320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.276139 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.276147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.276646 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.276650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.276930 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.277511 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.277533 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.279134 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.281124 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.281131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.281139 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.281590 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.281931 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.281934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.281936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.282081 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.282498 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.282505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.282751 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.282970 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.283278 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":84,"spinlock_wait_cycles":3584,"thread_start_us":105,"threads_started":1}
I20260806 11:42:03.284142 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.272096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.0.113.60:46597)
0806 11:42:03.272550 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.272570 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.272605 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:42:03.273277 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.273294 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.273534 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.273699 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.274474 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.275456 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.276259 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.276701 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.276931 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.279720 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.279725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.279734 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.279755 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.281610 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.282099 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.282105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.282107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.282192 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.282484 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.282488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.282490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.282733 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.282972 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.283195 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.283329 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":352,"spinlock_wait_cycles":56704,"thread_start_us":205,"threads_started":1}
I20260806 11:42:03.286109 10283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:42:03.287428 9510 heartbeater.cc:503] Master 127.0.113.62:44983 was elected leader, sending a full tablet report...
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.60@KRBTEST.COM for oryx/127.0.113.61@KRBTEST.COM
I20260806 11:42:03.294179 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.286536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.0.1:52410)
0806 11:42:03.286655 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.286687 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.286760 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.287189 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.287192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.287365 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.287454 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.287459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.288552 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.288556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.289132 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.289141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.289550 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.289552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.289622 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.289981 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.289992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.291097 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.293084 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.293086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.293087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.293251 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.293480 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.293482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.293483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.293549 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.293865 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.293867 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.293931 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.294002 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.294085 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 11:42:03.294301 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.286722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.0.113.61:39161)
0806 11:42:03.286983 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.286987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.286999 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:42:03.287033 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.287036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.287074 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.287176 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.287579 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.288398 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.289237 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.289455 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.289536 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.291221 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.291224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.291226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.291249 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.292994 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.293331 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.293333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.293334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.293405 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.293619 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.293621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.293622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.293770 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.293869 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.294031 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.294124 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260806 11:42:03.295082 10283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.60@KRBTEST.COM for oryx/127.0.113.60@KRBTEST.COM
I20260806 11:42:03.302338 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.295565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.0.1:57488)
0806 11:42:03.295650 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.295664 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.295734 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.295933 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.295935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.296054 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.296115 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.296119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.296835 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.296838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.297292 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.297299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.297702 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.297705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.297765 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.298278 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.298290 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.299423 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.300933 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.300936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.300937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.301141 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.301535 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.301537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.301539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.301616 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.301958 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.301960 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.302026 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.302103 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.302186 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 11:42:03.302341 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.295604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.0.113.60:46597)
0806 11:42:03.295709 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.295712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.295723 (+ 11us) server_negotiation.cc:407] Connection header received
0806 11:42:03.295818 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.295820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.295872 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.295926 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.296204 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.296702 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.297386 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.297589 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.297662 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.299537 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.299540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.299542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.299562 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.300829 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.301280 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.301282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.301283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.301383 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.301694 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.301696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.301697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.301858 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.301950 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.302159 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.302239 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 11:42:03.305768 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Bootstrap starting.
I20260806 11:42:03.307961 10283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:03.308743 10283 log.cc:824] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:03.310645 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: No bootstrap required, opened a new log
I20260806 11:42:03.313470 10283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } }
I20260806 11:42:03.313643 10283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:03.313715 10283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b6cc576f2a45cfbcd7a26633595361, State: Initialized, Role: FOLLOWER
I20260806 11:42:03.314092 10283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } }
I20260806 11:42:03.314512 10289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } } }
I20260806 11:42:03.314657 10289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:42:03.315210 10283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 11:42:03.319857 10307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:42:03.319947 10307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:42:03.323323 10051 master_runner.cc:428] Detected that this master 55b6cc576f2a45cfbcd7a26633595361 is joining an existing cluster
I20260806 11:42:03.323370 10051 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.113.60:46597
I20260806 11:42:03.334916 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.325721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.0.113.62:44983)
0806 11:42:03.325963 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.325970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.326471 (+ 501us) server_negotiation.cc:407] Connection header received
0806 11:42:03.326656 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.326659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.326705 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.326758 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.327366 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.328175 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.329876 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.330208 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.330295 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.331623 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.331626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.331628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.331650 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.333193 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.333666 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.333669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.333670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.333745 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.334057 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.334062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.334064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.334263 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.334343 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.334648 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.334797 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260806 11:42:03.334937 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.326054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:59696)
0806 11:42:03.326386 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.326400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.326548 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.326899 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.326902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.327092 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.327211 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.327217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.328358 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.328361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.329725 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.329733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.330451 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.330454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.330554 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.330970 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.330985 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.331473 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.333306 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.333309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.333311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.333563 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.333832 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.333835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.333837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.333941 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.334384 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.334387 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.334486 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.334609 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.334744 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":90,"thread_start_us":97,"threads_started":1}
I20260806 11:42:03.337391 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 9317
I20260806 11:42:03.355460 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:33483
--local_ip_for_outbound_sockets=127.0.113.1
--tserver_master_addrs=127.0.113.62:44983
--webserver_port=43013
--webserver_interface=127.0.113.1
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab
--principal=oryx/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.113.62:44983,127.0.113.61:39161
--tserver_master_addrs=127.0.113.62:44983,127.0.113.61:39161,127.0.113.60:46597 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016521, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.62@KRBTEST.COM for oryx/127.0.113.60@KRBTEST.COM
I20260806 11:42:03.364248 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.325834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.0.113.60:46597)
0806 11:42:03.325925 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.325928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.325940 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:42:03.325971 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.325975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.326018 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.326079 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.326785 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.327539 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.329054 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.329411 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.329521 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.330977 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.330980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.330982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.331002 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.362285 (+ 31283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.363052 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.363055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.363057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.363130 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.363381 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.363384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.363386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.363569 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.363656 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.363942 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.364080 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":28}
I20260806 11:42:03.364704 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.325391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.0.1:57498)
0806 11:42:03.325655 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.325671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.325762 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.326180 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.326183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.326477 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.326667 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.326675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.327672 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.327675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.328920 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.328931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.329542 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.329546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.329665 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.330284 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.330296 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.330646 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.362435 (+ 31789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.362439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.362441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.362951 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.363217 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.363219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.363221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.363288 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.363713 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.363715 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.363788 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.364412 (+ 624us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.364545 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":262,"thread_start_us":103,"threads_started":1}
I20260806 11:42:03.368553 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.353842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.0.1:57514)
0806 11:42:03.354006 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.354022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.354111 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.354866 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.354879 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.355114 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.355229 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.355239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.356262 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.356269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.357152 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.357162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.357642 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.357646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.357767 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.358351 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.358366 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.359859 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.366402 (+ 6543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.366406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.366408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.366930 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.367527 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.367531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.367533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.367624 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.368038 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.368041 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.368136 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.368214 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.368389 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 11:42:03.369177 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.354122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.0.113.60:46597)
0806 11:42:03.354436 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.354444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.354457 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:42:03.354531 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.354534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.354586 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.354831 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.355356 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.356134 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.357282 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.357514 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.357618 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.360031 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.360038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.360044 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.360077 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.366258 (+ 6181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.367314 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.367319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.367322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.367410 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.367734 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.367737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.367738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.367926 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.368827 (+ 901us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.368901 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.369012 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":126,"threads_started":1}
I20260806 11:42:03.370374 9215 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.113.60:46597: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361"
member_type: NON_VOTER
last_known_addr {
host: "127.0.113.60"
port: 46597
}
attrs {
promote: true
}
}
dest_uuid: "2fac143e703c4cb19d4b5ca844af0183"
cas_config_opid_index: 5
I20260806 11:42:03.370749 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.325346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.0.113.61:39161)
0806 11:42:03.325456 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.325461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.325854 (+ 393us) server_negotiation.cc:407] Connection header received
0806 11:42:03.326027 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.326029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.326081 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.326126 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.326802 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.327565 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.329064 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.329412 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.329510 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.366113 (+ 36603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.366118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.366120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.366159 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.368032 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.368941 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.368945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.368947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.369052 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.369576 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.369579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.369581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.369819 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.369914 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.370329 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.370503 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 11:42:03.371001 9215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } }
I20260806 11:42:03.371599 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.325473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.0.1:52418)
0806 11:42:03.325779 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.325792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.325921 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.326257 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.326259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.326518 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.326676 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.326681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.327690 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.327695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.328921 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.328931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.329662 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.329665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.329822 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.363700 (+ 33878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.363722 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.364234 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.368179 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.368185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.368189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.368505 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.369173 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.369175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.369177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.369461 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.369980 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.369985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.370091 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.371265 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.371423 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":236,"thread_start_us":97,"threads_started":1}
I20260806 11:42:03.372459 10043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 55b6cc576f2a45cfbcd7a26633595361 (127.0.113.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.373898 10043 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.113.60:46597
I20260806 11:42:03.374049 10043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.374142 10043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:03.374737 10051 master.cc:537] Master@127.0.113.60:46597 shutting down...
I20260806 11:42:03.375273 10243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:03.376964 10243 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Refusing update from remote peer 2fac143e703c4cb19d4b5ca844af0183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 11:42:03.378619 10041 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:03.380048 10289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:42:03.380194 10289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } }
I20260806 11:42:03.381389 10289 leader_election.cc:288] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fac143e703c4cb19d4b5ca844af0183 (127.0.113.62:44983), f78d76dabc0946e7a419656c7bac1a81 (127.0.113.61:39161)
I20260806 11:42:03.381575 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.374506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.0.113.61:39161)
0806 11:42:03.374608 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.374612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.377356 (+ 2744us) server_negotiation.cc:407] Connection header received
0806 11:42:03.377403 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.377405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.377462 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.377548 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:03.377895 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.378699 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.380232 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.380870 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.380976 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.381227 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.381350 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.381443 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 11:42:03.381906 10119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55b6cc576f2a45cfbcd7a26633595361" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78d76dabc0946e7a419656c7bac1a81" is_pre_election: true
I20260806 11:42:03.381982 10051 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:03.382072 10119 raft_consensus.cc:1799] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 LEARNER]: Handling vote request from an unknown peer 55b6cc576f2a45cfbcd7a26633595361
I20260806 11:42:03.382191 10119 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 55b6cc576f2a45cfbcd7a26633595361 for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260806 11:42:03.382194 9236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55b6cc576f2a45cfbcd7a26633595361" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fac143e703c4cb19d4b5ca844af0183" is_pre_election: true
I20260806 11:42:03.382251 10051 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:03.382344 10051 tablet_replica.cc:330] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: stopping tablet replica
I20260806 11:42:03.382710 10209 leader_election.cc:302] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55b6cc576f2a45cfbcd7a26633595361; no voters: 2fac143e703c4cb19d4b5ca844af0183, f78d76dabc0946e7a419656c7bac1a81
W20260806 11:42:03.382858 9202 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 -> Peer 55b6cc576f2a45cfbcd7a26633595361 (127.0.113.60:46597): Couldn't send request to peer 55b6cc576f2a45cfbcd7a26633595361. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
W20260806 11:42:03.382911 10209 raft_consensus.cc:2670] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260806 11:42:03.383462 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.374415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.0.1:52422)
0806 11:42:03.374727 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.374740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.374810 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.377660 (+ 2850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.377663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.377680 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:03.377778 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.377783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.378834 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.378838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.380090 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.380103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.380996 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.381000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.381088 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.381216 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.383265 (+ 2049us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.383346 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260806 11:42:03.383983 10119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 LEARNER]: Refusing update from remote peer 2fac143e703c4cb19d4b5ca844af0183: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 11:42:03.384624 10322 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 11:42:03.386576 10119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 55b6cc576f2a45cfbcd7a26633595361 (127.0.113.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.388761 10051 master.cc:552] Master@127.0.113.60:46597 shutdown complete.
I20260806 11:42:03.388813 10051 master_runner.cc:305] Clearing existing system tablet
I20260806 11:42:03.389107 10051 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:03.389176 10051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:03.389225 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fac143e703c4cb19d4b5ca844af0183. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.389343 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: This master's current role is: LEARNER
I20260806 11:42:03.389458 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: NON_VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: true } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.389550 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: This master's current role is: LEARNER
I20260806 11:42:03.390440 10051 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:03.391359 10326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.391634 10051 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:03.391788 10051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:03.391825 10051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal
uuid: "55b6cc576f2a45cfbcd7a26633595361"
format_stamp: "Formatted at 2026-08-06 11:42:03 on dist-test-slave-45dx"
server_key: "7e6f197e65713c383ed6afb3b90c226a"
server_key_iv: "41cc06d9292fa4e2f5ab47c31a1f5b4b"
server_key_version: "encryptionkey@0"
I20260806 11:42:03.393476 10051 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:03.395289 10051 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 11:42:03.395393 10051 log.cc:1196] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 11:42:03.395648 10051 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Deleting consensus metadata
I20260806 11:42:03.395828 10051 master_runner.cc:315] Copying system tablet from 127.0.113.62:44983
I20260806 11:42:03.397179 10322 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: attempting to promote NON_VOTER f78d76dabc0946e7a419656c7bac1a81 to VOTER
I20260806 11:42:03.397238 10051 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: tablet copy: Beginning tablet copy session from remote peer at address 127.0.113.62:44983
I20260806 11:42:03.397694 10322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } }
I20260806 11:42:03.399425 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.399005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.0.1:57522)
0806 11:42:03.399231 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.399329 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260806 11:42:03.399966 10119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 LEARNER]: Refusing update from remote peer 2fac143e703c4cb19d4b5ca844af0183: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 11:42:03.400740 10043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 11:42:03.401726 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.401386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.0.1:57538)
0806 11:42:03.401594 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.401648 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260806 11:42:03.402508 9202 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 -> Peer 55b6cc576f2a45cfbcd7a26633595361 (127.0.113.60:46597): Couldn't send request to peer 55b6cc576f2a45cfbcd7a26633595361. Status: Network error: Client connection negotiation failed: client connection to 127.0.113.60:46597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 11:42:03.403191 10043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f78d76dabc0946e7a419656c7bac1a81 (127.0.113.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.404970 10043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.405078 10043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:03.405238 10043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.405303 10043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:03.405994 10119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f78d76dabc0946e7a419656c7bac1a81 (127.0.113.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.407770 10324 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.407928 10324 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:42:03.408212 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.397858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.0.113.62:44983)
0806 11:42:03.397978 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.397984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.398520 (+ 536us) server_negotiation.cc:407] Connection header received
0806 11:42:03.398712 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.398717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.398763 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.398817 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.399527 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.400259 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.401443 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.401757 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.401856 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.403456 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.403460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.403462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.403486 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.406127 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.406735 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.406738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.406740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.406820 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.407127 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.407130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.407132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.407385 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.407461 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.407940 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.408076 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:42:03.408854 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.398074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:59710)
0806 11:42:03.398431 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.398446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.398599 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.398994 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.398997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.399245 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.399388 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.399397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.400434 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.400437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.401318 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.401329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.402064 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.402067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.402165 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.402837 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.402856 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.403306 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.406321 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.406328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.406333 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.406579 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.406931 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.406934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.406936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.407022 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.407537 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.407539 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.407639 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.408587 (+ 948us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.408728 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":88,"threads_started":1}
I20260806 11:42:03.409437 9253 tablet_copy_service.cc:140] P 2fac143e703c4cb19d4b5ca844af0183: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 55b6cc576f2a45cfbcd7a26633595361 ({username='oryx', principal='oryx/127.0.113.60@KRBTEST.COM'} at 127.0.0.1:59710)
I20260806 11:42:03.409526 9253 tablet_copy_service.cc:161] P 2fac143e703c4cb19d4b5ca844af0183: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 55b6cc576f2a45cfbcd7a26633595361 at {username='oryx', principal='oryx/127.0.113.60@KRBTEST.COM'} at 127.0.0.1:59710: session id = 55b6cc576f2a45cfbcd7a26633595361-00000000000000000000000000000000
I20260806 11:42:03.410311 9253 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 11:42:03.411397 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 11:42:03.413391 10051 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: tablet copy: Starting download of 0 data blocks...
I20260806 11:42:03.413582 10051 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: tablet copy: Starting download of 1 WAL segments...
I20260806 11:42:03.414861 10051 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 11:42:03.416653 9253 tablet_copy_service.cc:342] P 2fac143e703c4cb19d4b5ca844af0183: Request end of tablet copy session 55b6cc576f2a45cfbcd7a26633595361-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.113.60@KRBTEST.COM'} at 127.0.0.1:59710
I20260806 11:42:03.416746 9253 tablet_copy_service.cc:433] P 2fac143e703c4cb19d4b5ca844af0183: ending tablet copy session 55b6cc576f2a45cfbcd7a26633595361-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 55b6cc576f2a45cfbcd7a26633595361
I20260806 11:42:03.423350 10051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:03.425007 10336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.425156 10338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.425246 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:03.425726 10051 server_base.cc:1034] running on GCE node
I20260806 11:42:03.425885 10051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:03.426235 10051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:03.427377 10051 hybrid_clock.cc:648] HybridClock initialized: now 1786016523427381 us; error 41 us; skew 500 ppm
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:03.428864 10051 init.cc:377] Logged in from keytab as oryx/127.0.113.60@KRBTEST.COM (short username oryx)
I20260806 11:42:03.429332 10051 webserver.cc:460] Webserver started at http://127.0.113.60:38383/ using document root <none> and password file <none>
I20260806 11:42:03.429510 10051 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:03.429563 10051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:03.430737 10051 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:03.431655 10352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.431921 10051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:03.431996 10051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal
uuid: "55b6cc576f2a45cfbcd7a26633595361"
format_stamp: "Formatted at 2026-08-06 11:42:03 on dist-test-slave-45dx"
server_key: "7e6f197e65713c383ed6afb3b90c226a"
server_key_iv: "41cc06d9292fa4e2f5ab47c31a1f5b4b"
server_key_version: "encryptionkey@0"
I20260806 11:42:03.432070 10051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:03.450969 10051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:03.453189 10051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:03.458392 10051 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.60:46597
I20260806 11:42:03.458390 10435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.60:46597 every 8 connection(s)
I20260806 11:42:03.459125 10051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
I20260806 11:42:03.469785 10436 sys_catalog.cc:263] Verifying existing consensus state
I20260806 11:42:03.470651 10436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Bootstrap starting.
I20260806 11:42:03.479331 10436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:42:03.479687 10436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Bootstrap complete.
I20260806 11:42:03.480172 10436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } }
I20260806 11:42:03.480312 10436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55b6cc576f2a45cfbcd7a26633595361, State: Initialized, Role: LEARNER
I20260806 11:42:03.480471 10436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } }
I20260806 11:42:03.480804 10439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.480965 10436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:42:03.480988 10439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: This master's current role is: LEARNER
I20260806 11:42:03.484676 10454 catalog_manager.cc:1259] Loaded cluster ID: 977f3e83e3b54ec4b71ace2eccabe00f
I20260806 11:42:03.484743 10454 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: loading cluster ID for follower catalog manager: success
I20260806 11:42:03.486451 10454 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: acquiring CA information for follower catalog manager: success
I20260806 11:42:03.487614 10454 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 11:42:03.511790 9714 heartbeater.cc:503] Master 127.0.113.62:44983 was elected leader, sending a full tablet report...
W20260806 11:42:03.520421 10321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:03.520750 10321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:03.520816 10321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:03.526939 10321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:03.527017 10321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:03.527096 10321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:42:03.532745 10321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab
--principal=oryx/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:33483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=43013
--tserver_master_addrs=127.0.113.62:44983,127.0.113.61:39161,127.0.113.60:46597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:03.533854 10321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:03.534955 10321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:03.544854 10462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.544870 10465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:03.545182 10321 server_base.cc:1034] running on GCE node
W20260806 11:42:03.545269 10461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:03.545738 10321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:03.546417 10321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:03.547628 10321 hybrid_clock.cc:648] HybridClock initialized: now 1786016523547607 us; error 29 us; skew 500 ppm
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:03.550602 10321 init.cc:377] Logged in from keytab as oryx/127.0.113.1@KRBTEST.COM (short username oryx)
I20260806 11:42:03.551545 10321 webserver.cc:460] Webserver started at http://127.0.113.1:43013/ using document root <none> and password file <none>
I20260806 11:42:03.552167 10321 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:03.552295 10321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:03.556648 10321 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:42:03.558842 10473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.560223 10321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 11:42:03.560424 10321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "dd8f1735feb8499f95df54292701beec"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "5f9068f36df5a16fcf4c6f8661f43993"
server_key_iv: "3316fd0ccfe1851343e56d624bed227d"
server_key_version: "encryptionkey@0"
I20260806 11:42:03.561004 10321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:03.574070 10321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:03.577066 10321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:03.577279 10321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:03.577903 10321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:03.578826 10321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:03.578879 10321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.578953 10321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:03.578989 10321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.589972 10321 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:33483
I20260806 11:42:03.589998 10632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:33483 every 8 connection(s)
I20260806 11:42:03.590960 10321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:42:03.594352 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 10321
I20260806 11:42:03.594579 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 9519
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.60@KRBTEST.COM
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.61@KRBTEST.COM
I20260806 11:42:03.610512 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.592962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:39889 (local address 127.0.113.60:46597)
0806 11:42:03.593453 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.593458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.594432 (+ 974us) server_negotiation.cc:407] Connection header received
0806 11:42:03.595819 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.595825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.595910 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.595988 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.597760 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.598698 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.600367 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.600714 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.600813 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.605441 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.605446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.605452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.605482 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.608093 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.608681 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.608687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.608692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.608771 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.609157 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.609162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.609166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.609391 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.609482 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.610136 (+ 654us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.610274 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":76,"threads_started":1}
I20260806 11:42:03.610675 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.593185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:53609 (local address 127.0.113.61:39161)
0806 11:42:03.593453 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.593459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.594448 (+ 989us) server_negotiation.cc:407] Connection header received
0806 11:42:03.595795 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.595801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.595874 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.595926 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.597799 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.598889 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.600594 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.601104 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.601218 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.605230 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.605239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.605245 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.605287 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.607726 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.608720 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.608727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.608732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.608867 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.609311 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.609318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.609323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.609634 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.609741 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.610251 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.610416 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260806 11:42:03.612123 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.593281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.113.1:39889)
0806 11:42:03.594337 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.594351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.595554 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.596327 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.596342 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.596915 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.597604 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.597624 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.598864 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.598878 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.600196 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.600209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.601102 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.601108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.601329 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.602642 (+ 1313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.602670 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.605311 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.608213 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.608219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.608223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.608581 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.608868 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.608882 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.608886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.609055 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.609587 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.609595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.609889 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.610708 (+ 819us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.611124 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":240,"spinlock_wait_cycles":12032,"thread_start_us":130,"threads_started":1}
I20260806 11:42:03.612133 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.593281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.113.1:53609)
0806 11:42:03.594359 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.594384 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.595557 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.596337 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.596342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.596896 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.597608 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.597631 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.599074 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.599080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.600415 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.600431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.601441 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.601448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.601591 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.602809 (+ 1218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.602844 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.605057 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.607921 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.607932 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.607954 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.608553 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.609013 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.609019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.609024 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.609167 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.609818 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.609824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.609985 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.610868 (+ 883us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.611247 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":382,"spinlock_wait_cycles":28928,"thread_start_us":110,"threads_started":1}
I20260806 11:42:03.613408 10633 heartbeater.cc:347] Connected to a master server at 127.0.113.60:46597
I20260806 11:42:03.613613 10633 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:03.614117 10633 heartbeater.cc:511] Master 127.0.113.60:46597 requested a full tablet report, sending...
I20260806 11:42:03.616032 10384 ts_manager.cc:194] Registered new tserver with Master: dd8f1735feb8499f95df54292701beec (127.0.113.1:33483)
I20260806 11:42:03.616576 10636 heartbeater.cc:347] Connected to a master server at 127.0.113.61:39161
I20260806 11:42:03.616673 10636 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:03.616837 10636 heartbeater.cc:511] Master 127.0.113.61:39161 requested a full tablet report, sending...
I20260806 11:42:03.618173 10102 ts_manager.cc:194] Registered new tserver with Master: dd8f1735feb8499f95df54292701beec (127.0.113.1:33483)
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:03.626886 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.593281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.1:35109)
0806 11:42:03.593832 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.593885 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.617055 (+ 23170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.617496 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.617498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.617728 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.617850 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.617857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.619134 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.619139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.619920 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.619930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.620625 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.620628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.620696 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.621214 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.621225 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.623249 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.625420 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.625422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.625423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.625623 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.625951 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.625953 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.625954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.626029 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.626468 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.626471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.626560 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.626646 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.626752 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":170,"threads_started":1}
I20260806 11:42:03.626850 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:33957
--local_ip_for_outbound_sockets=127.0.113.2
--tserver_master_addrs=127.0.113.62:44983
--webserver_port=44295
--webserver_interface=127.0.113.2
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab
--principal=oryx/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.113.62:44983,127.0.113.61:39161
--tserver_master_addrs=127.0.113.62:44983,127.0.113.61:39161,127.0.113.60:46597 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:42:03.627079 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.593338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:35109 (local address 127.0.113.62:44983)
0806 11:42:03.593464 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.593469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.593986 (+ 517us) server_negotiation.cc:407] Connection header received
0806 11:42:03.617214 (+ 23228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.617219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.617281 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.617339 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.617988 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.619003 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.620045 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.620483 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.620603 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.623403 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.623410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.623416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.623453 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.625302 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.625732 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.625739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.625743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.625843 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.626142 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.626146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.626148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.626332 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.626462 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.626733 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.626894 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 11:42:03.627951 10634 heartbeater.cc:347] Connected to a master server at 127.0.113.62:44983
I20260806 11:42:03.628052 10634 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:03.628265 10634 heartbeater.cc:511] Master 127.0.113.62:44983 requested a full tablet report, sending...
I20260806 11:42:03.629145 9215 ts_manager.cc:194] Re-registered known tserver with Master: dd8f1735feb8499f95df54292701beec (127.0.113.1:33483)
I20260806 11:42:03.629724 9215 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:35109
W20260806 11:42:03.739485 10645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:03.739709 10645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:03.739784 10645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 11:42:03.743063 9898 heartbeater.cc:503] Master 127.0.113.62:44983 was elected leader, sending a full tablet report...
W20260806 11:42:03.744442 10645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:03.744514 10645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:03.744587 10645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:42:03.749738 10645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab
--principal=oryx/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:33957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=44295
--tserver_master_addrs=127.0.113.62:44983,127.0.113.61:39161,127.0.113.60:46597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:03.750659 10645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:03.751564 10645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:03.760856 10655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.760871 10652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.760874 10651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:03.761515 10645 server_base.cc:1034] running on GCE node
I20260806 11:42:03.761915 10645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:03.762502 10645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:03.763688 10645 hybrid_clock.cc:648] HybridClock initialized: now 1786016523763678 us; error 33 us; skew 500 ppm
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:03.766569 10645 init.cc:377] Logged in from keytab as oryx/127.0.113.2@KRBTEST.COM (short username oryx)
I20260806 11:42:03.767378 10645 webserver.cc:460] Webserver started at http://127.0.113.2:44295/ using document root <none> and password file <none>
I20260806 11:42:03.767921 10645 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:03.768026 10645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:03.771633 10645 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:03.773659 10662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.774803 10645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:03.774955 10645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "6850047bf5d34a4fa628dd2366429165"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "e2f7d0e920d573ea88ae01fd428aca70"
server_key_iv: "bb5270683acd858b04b6aad9ec2d5c4a"
server_key_version: "encryptionkey@0"
I20260806 11:42:03.775463 10645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:03.795629 10645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:03.798390 10645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:03.798584 10645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:03.799149 10645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:03.799973 10645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:03.800045 10645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.800118 10645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:03.800171 10645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.807389 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.803867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.113.60:46597)
0806 11:42:03.803994 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.804001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.804046 (+ 45us) server_negotiation.cc:407] Connection header received
0806 11:42:03.804263 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.804267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.804387 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.804498 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:03.804774 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.805392 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.806402 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.806845 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.806968 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.807117 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.807190 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.807254 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 11:42:03.807389 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.803757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.0.1:57554)
0806 11:42:03.803978 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.803993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.804099 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.804516 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.804520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.804541 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:03.804644 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.804652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.805515 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.805520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.806267 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.806276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.806974 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.806978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.807067 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.807089 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.807166 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.807250 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260806 11:42:03.808112 10390 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 11:42:03.808622 10439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fac143e703c4cb19d4b5ca844af0183. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:03.808794 10439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: This master's current role is: LEARNER
I20260806 11:42:03.811326 10645 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:33957
I20260806 11:42:03.811364 10826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:33957 every 8 connection(s)
I20260806 11:42:03.812288 10645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:42:03.815595 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 10645
I20260806 11:42:03.815780 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 9720
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.60@KRBTEST.COM
I20260806 11:42:03.835449 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:43973
--local_ip_for_outbound_sockets=127.0.113.3
--tserver_master_addrs=127.0.113.62:44983
--webserver_port=36805
--webserver_interface=127.0.113.3
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab
--principal=oryx/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.113.62:44983,127.0.113.61:39161
--tserver_master_addrs=127.0.113.62:44983,127.0.113.61:39161,127.0.113.60:46597 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.61@KRBTEST.COM
I20260806 11:42:03.841285 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.814757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:48553 (local address 127.0.113.60:46597)
0806 11:42:03.814866 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.814871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.826975 (+ 12104us) server_negotiation.cc:407] Connection header received
0806 11:42:03.827138 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.827141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.827190 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.827239 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.829137 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.829925 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.832423 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.832730 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.832807 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.836040 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.836047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.836052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.836085 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.838635 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.839402 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.839408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.839413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.839494 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.839945 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.839948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.839950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.840180 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.840260 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.841002 (+ 742us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.841122 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:42:03.841689 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.813993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:42881 (local address 127.0.113.61:39161)
0806 11:42:03.814116 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.814121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.825895 (+ 11774us) server_negotiation.cc:407] Connection header received
0806 11:42:03.826896 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.826901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.826970 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.827026 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.829138 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.829935 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.832043 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.832399 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.832475 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.837370 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.837376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.837380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.837413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.839414 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.839947 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.839954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.839958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.840050 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.840498 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.840504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.840509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.840767 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.840870 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.841304 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.841440 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 11:42:03.842293 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.814677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.113.2:48553)
0806 11:42:03.826896 (+ 12219us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.826910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.827036 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.827377 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.827380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.828090 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.828966 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.828986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.830114 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.830118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.832273 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.832283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.832975 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.832978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.833073 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.833800 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.833813 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.835861 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.838771 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.838777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.838794 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.839255 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.839621 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.839624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.839625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.839823 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.840453 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.840456 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.840720 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.841452 (+ 732us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.841614 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12156,"mutex_wait_us":467,"spinlock_wait_cycles":17408,"thread_start_us":119,"threads_started":1}
I20260806 11:42:03.842474 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.814453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.113.2:42881)
0806 11:42:03.825790 (+ 11337us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.825808 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.826734 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.827346 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.827358 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.828085 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.828965 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.828986 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.830114 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.830118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.831904 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.831915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.832657 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.832661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.832970 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.833655 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.833696 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.837175 (+ 3479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.839534 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.839540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.839545 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.839825 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.840161 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.840167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.840172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.840266 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.840944 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.840950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.841090 (+ 140us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.841950 (+ 860us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.842128 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11226,"mutex_wait_us":496,"spinlock_wait_cycles":22400,"thread_start_us":139,"threads_started":1}
I20260806 11:42:03.842566 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.814452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.2:49503)
0806 11:42:03.815101 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:03.815160 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:03.826486 (+ 11326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:03.827347 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:03.827359 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:03.828061 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:03.828963 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.828987 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.830236 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.830242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.831886 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:03.831900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.832657 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.832664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:03.832937 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.833975 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:03.833995 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:03.836339 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:03.838914 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.838917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.838919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.839241 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.839612 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:03.839615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:03.839617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:03.839824 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:03.840370 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:03.840377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:03.840693 (+ 316us) client_negotiation.cc:769] Sending connection context
0806 11:42:03.840913 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 11:42:03.841232 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":345,"spinlock_wait_cycles":72064,"thread_start_us":135,"threads_started":1}
I20260806 11:42:03.843751 10830 heartbeater.cc:347] Connected to a master server at 127.0.113.61:39161
I20260806 11:42:03.843750 10827 heartbeater.cc:347] Connected to a master server at 127.0.113.60:46597
I20260806 11:42:03.843976 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:03.815463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:49503 (local address 127.0.113.62:44983)
0806 11:42:03.815580 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:03.815584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:03.815596 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:42:03.826772 (+ 11176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:03.826779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:03.826844 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:03.826914 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:03.829137 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.830068 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.832039 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:03.832380 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:03.832458 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:03.836518 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:03.836524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:03.836530 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:03.836564 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:03.838785 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.839351 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.839355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.839359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.839460 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:03.839951 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:03.839954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:03.839957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:03.840171 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:03.840260 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:03.843676 (+ 3416us) server_negotiation.cc:299] Negotiation successful
0806 11:42:03.843812 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:42:03.844049 10830 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:03.844049 10827 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:03.844692 10827 heartbeater.cc:511] Master 127.0.113.60:46597 requested a full tablet report, sending...
I20260806 11:42:03.845194 10828 heartbeater.cc:347] Connected to a master server at 127.0.113.62:44983
I20260806 11:42:03.845278 10828 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:03.845448 10828 heartbeater.cc:511] Master 127.0.113.62:44983 requested a full tablet report, sending...
I20260806 11:42:03.845659 10830 heartbeater.cc:511] Master 127.0.113.61:39161 requested a full tablet report, sending...
I20260806 11:42:03.845886 10384 ts_manager.cc:194] Registered new tserver with Master: 6850047bf5d34a4fa628dd2366429165 (127.0.113.2:33957)
I20260806 11:42:03.846024 9215 ts_manager.cc:194] Re-registered known tserver with Master: 6850047bf5d34a4fa628dd2366429165 (127.0.113.2:33957)
I20260806 11:42:03.846698 9215 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:49503
I20260806 11:42:03.846782 10102 ts_manager.cc:194] Registered new tserver with Master: 6850047bf5d34a4fa628dd2366429165 (127.0.113.2:33957)
W20260806 11:42:03.950248 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:03.950527 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:03.950625 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:03.954633 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:03.954712 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:03.954797 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:42:03.959915 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab
--principal=oryx/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:43973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=36805
--tserver_master_addrs=127.0.113.62:44983,127.0.113.61:39161,127.0.113.60:46597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:03.961015 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:03.962023 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:03.971562 10848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.971622 10843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:03.971585 10844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:03.972030 10836 server_base.cc:1034] running on GCE node
I20260806 11:42:03.972496 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:03.973097 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:03.974282 10836 hybrid_clock.cc:648] HybridClock initialized: now 1786016523974267 us; error 31 us; skew 500 ppm
Aug 06 11:42:03 dist-test-slave-45dx krb5kdc[9159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:03.976964 10836 init.cc:377] Logged in from keytab as oryx/127.0.113.3@KRBTEST.COM (short username oryx)
I20260806 11:42:03.977792 10836 webserver.cc:460] Webserver started at http://127.0.113.3:36805/ using document root <none> and password file <none>
I20260806 11:42:03.978407 10836 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:03.978521 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:03.982717 10836 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:42:03.984913 10856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:03.986143 10836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:03.986284 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "11654e1660a64ad7a09ed46610715c1b"
format_stamp: "Formatted at 2026-08-06 11:42:02 on dist-test-slave-45dx"
server_key: "dccdb16ae2dc8487e345fbc78637c9c8"
server_key_iv: "391c20447a7710dd73c0abfdec3a9cfa"
server_key_version: "encryptionkey@0"
I20260806 11:42:03.986815 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:04.028169 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:04.031042 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:04.031268 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:04.031888 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:04.032917 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:04.032989 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:04.033056 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:04.033105 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:04.042534 10836 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:43973
I20260806 11:42:04.042559 11023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:43973 every 8 connection(s)
I20260806 11:42:04.043545 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:42:04.044209 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 10836
I20260806 11:42:04.057782 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.047089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.0.113.61:39161)
0806 11:42:04.047211 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.047215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.047673 (+ 458us) server_negotiation.cc:407] Connection header received
0806 11:42:04.047834 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.047837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.047891 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.047975 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.049145 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.049945 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.051137 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.051480 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.051548 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.053714 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.053722 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.053727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.053758 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.055708 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.056248 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.056254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.056257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.056373 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.056734 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.056740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.056743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.056997 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.057101 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.057462 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.057584 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 11:42:04 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
Aug 06 11:42:04 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.60@KRBTEST.COM
I20260806 11:42:04.059399 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.046280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.0.113.62:44983)
0806 11:42:04.047971 (+ 1691us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.047975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.048238 (+ 263us) server_negotiation.cc:407] Connection header received
0806 11:42:04.048449 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.048452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.048511 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.048582 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.049555 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.050316 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.051383 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.052272 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.052400 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.053971 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.053977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.053981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.054010 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.056985 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.057466 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.057470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.057472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.057596 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.057917 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.057921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.057922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.058141 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.058257 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.059101 (+ 844us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.059227 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":473,"server-negotiator.queue_time_us":1596,"thread_start_us":99,"threads_started":1}
Aug 06 11:42:04 dist-test-slave-45dx krb5kdc[9159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016523, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.61@KRBTEST.COM
I20260806 11:42:04.063076 9215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 11:42:04.067018 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.045527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:40119 (local address 127.0.113.62:44983)
0806 11:42:04.045701 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.045706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.047836 (+ 2130us) server_negotiation.cc:407] Connection header received
0806 11:42:04.049289 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.049293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.049350 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.049423 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.051508 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.052486 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.054731 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.055168 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.055257 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.060249 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.060254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.060256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.060282 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.062626 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.065207 (+ 2581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.065211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.065213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.065292 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.065735 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.065739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.065741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.065970 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.066059 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.066742 (+ 683us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.066869 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":70}
I20260806 11:42:04.067329 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.047362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:57825 (local address 127.0.113.60:46597)
0806 11:42:04.047794 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.047800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.048483 (+ 683us) server_negotiation.cc:407] Connection header received
0806 11:42:04.049397 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.049401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.049452 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.049531 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.051509 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.052486 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.055453 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.055837 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.055943 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.062989 (+ 7046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.062993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.062996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.063022 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.065083 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.065856 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.065861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.065865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.065982 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.066352 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.066357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.066359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.066590 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.066664 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.067031 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.067168 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":98,"threads_started":1}
W20260806 11:42:04.067849 9215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:42:04.068847 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.045923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.3:40119)
0806 11:42:04.047682 (+ 1759us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:04.047740 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:04.049130 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:04.049738 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:04.049747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:04.050363 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:04.051338 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.051360 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.052674 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.052678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.054560 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.054573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.055462 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.055467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.055676 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.056294 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:04.056377 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:04.059525 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:04.062925 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:04.062932 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:04.062948 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:04.063376 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:04.065409 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:04.065413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:04.065415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:04.065611 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:04.066150 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:04.066157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:04.066463 (+ 306us) client_negotiation.cc:769] Sending connection context
0806 11:42:04.067202 (+ 739us) client_negotiation.cc:240] Negotiation successful
0806 11:42:04.067640 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1561,"mutex_wait_us":321,"spinlock_wait_cycles":78592,"thread_start_us":784,"threads_started":1}
I20260806 11:42:04.069448 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.046583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.0.113.60:46597)
0806 11:42:04.046727 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.046734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.048736 (+ 2002us) server_negotiation.cc:407] Connection header received
0806 11:42:04.048925 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.048929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.048985 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.049038 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.056574 (+ 7536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.057403 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.059529 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.059885 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.059968 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.061877 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.061883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.061889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.061927 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.064100 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.068019 (+ 3919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.068023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.068025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.068159 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.068500 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.068503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.068504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.068698 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.068811 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.069168 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.069303 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:42:04.069885 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.047579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:42303 (local address 127.0.113.61:39161)
0806 11:42:04.057994 (+ 10415us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.057998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.058010 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:42:04.058041 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.058044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.058081 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.058154 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.060136 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.060888 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.061973 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.062334 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.062422 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.065180 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.065184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.065187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.065212 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.067001 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.068026 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.068029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.068032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.068127 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.068497 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.068500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.068502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.068698 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.068770 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.069414 (+ 644us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.069727 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10350,"thread_start_us":92,"threads_started":1}
I20260806 11:42:04.070240 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.047107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.113.3:42303)
0806 11:42:04.047937 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:04.047949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:04.048941 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:04.059679 (+ 10738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:04.059681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:04.059861 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:04.059982 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.059989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.061034 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.061037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.061805 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.061836 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.062594 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.062599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.062710 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.063550 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:04.063566 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:04.065039 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:04.067547 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:04.067552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:04.067556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:04.067812 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:04.068252 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:04.068255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:04.068257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:04.068368 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:04.068849 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:04.068853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:04.069210 (+ 357us) client_negotiation.cc:769] Sending connection context
0806 11:42:04.069884 (+ 674us) client_negotiation.cc:240] Negotiation successful
0806 11:42:04.070058 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"spinlock_wait_cycles":10240,"thread_start_us":161,"threads_started":1}
I20260806 11:42:04.070506 11025 heartbeater.cc:347] Connected to a master server at 127.0.113.62:44983
I20260806 11:42:04.070773 11025 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:04.070945 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.047173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.113.3:57825)
0806 11:42:04.048356 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:04.048369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:04.049289 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:04.049738 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:04.049747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:04.050363 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:04.051338 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.051360 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.052677 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.052680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.055298 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.055309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.056152 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.056159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.056279 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.057573 (+ 1294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:04.057594 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:04.062795 (+ 5201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:04.065458 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:04.065462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:04.065466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:04.065730 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:04.066130 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:04.066135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:04.066138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:04.066233 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:04.066740 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:04.066745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:04.066838 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 11:42:04.067472 (+ 634us) client_negotiation.cc:240] Negotiation successful
0806 11:42:04.069314 (+ 1842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":2757,"spinlock_wait_cycles":8704,"thread_start_us":103,"threads_started":1}
I20260806 11:42:04.070958 11027 heartbeater.cc:347] Connected to a master server at 127.0.113.61:39161
I20260806 11:42:04.071071 11027 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:04.071398 11027 heartbeater.cc:511] Master 127.0.113.61:39161 requested a full tablet report, sending...
I20260806 11:42:04.071731 11024 heartbeater.cc:347] Connected to a master server at 127.0.113.60:46597
I20260806 11:42:04.071834 11024 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:04.072002 11024 heartbeater.cc:511] Master 127.0.113.60:46597 requested a full tablet report, sending...
I20260806 11:42:04.072465 10322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: NON_VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: true } } }
I20260806 11:42:04.072570 10322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:04.072587 10102 ts_manager.cc:194] Registered new tserver with Master: 11654e1660a64ad7a09ed46610715c1b (127.0.113.3:43973)
I20260806 11:42:04.072750 10384 ts_manager.cc:194] Registered new tserver with Master: 11654e1660a64ad7a09ed46610715c1b (127.0.113.3:43973)
I20260806 11:42:04.071398 11025 heartbeater.cc:511] Master 127.0.113.62:44983 requested a full tablet report, sending...
I20260806 11:42:04.074165 9217 ts_manager.cc:194] Re-registered known tserver with Master: 11654e1660a64ad7a09ed46610715c1b (127.0.113.3:43973)
I20260806 11:42:04.074868 9217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:40119
I20260806 11:42:04.078796 11055 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: attempting to promote NON_VOTER 55b6cc576f2a45cfbcd7a26633595361 to VOTER
I20260806 11:42:04.079162 11055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } }
I20260806 11:42:04.080952 10118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Refusing update from remote peer 2fac143e703c4cb19d4b5ca844af0183: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260806 11:42:04.081348 11054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 11:42:04.081441 10386 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 LEARNER]: Refusing update from remote peer 2fac143e703c4cb19d4b5ca844af0183: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260806 11:42:04.082494 11054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 11:42:04.082993 10322 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 55b6cc576f2a45cfbcd7a26633595361 (127.0.113.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:04.083547 10118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 55b6cc576f2a45cfbcd7a26633595361 (127.0.113.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:04.084971 10322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:04.085094 10322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:04.085210 10322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:04.085294 10322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:04.085417 10322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:04.085508 10322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:04.085184 10386 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 55b6cc576f2a45cfbcd7a26633595361 (127.0.113.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:04.087512 11073 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:04.087678 11073 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:42:04.087599 10439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fac143e703c4cb19d4b5ca844af0183" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:04.087733 10439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:42:04.091892 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.085168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:33483 (local address 127.0.0.1:38842)
0806 11:42:04.086584 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:04.086600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:04.086806 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:04.087522 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:04.087526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:04.087547 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:04.087645 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.087695 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.089143 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.089149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.090263 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.090274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.091356 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.091361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.091467 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.091494 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 11:42:04.091671 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 11:42:04.091751 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330}
I20260806 11:42:04.091950 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.085702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.0.113.1:33483)
0806 11:42:04.085972 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.085989 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.086704 (+ 715us) server_negotiation.cc:407] Connection header received
0806 11:42:04.086953 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.086959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.087184 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.087344 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:04.087895 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.089028 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.090419 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.091196 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.091322 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.091490 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.091704 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.091802 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260806 11:42:04.094249 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.086633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:33957 (local address 127.0.0.1:50294)
0806 11:42:04.088926 (+ 2293us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:04.088942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:04.089048 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:04.089802 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:04.089805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:04.089825 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:04.089942 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.089952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.091514 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.091518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.092888 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.092897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.093901 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.093905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.094003 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.094027 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 11:42:04.094068 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 11:42:04.094129 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2208,"thread_start_us":95,"threads_started":1}
I20260806 11:42:04.094892 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.087076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.0.113.2:33957)
0806 11:42:04.089339 (+ 2263us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.089355 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.089382 (+ 27us) server_negotiation.cc:407] Connection header received
0806 11:42:04.089476 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.089481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.089654 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.089840 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:04.090093 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.091384 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.093005 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.093759 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.094409 (+ 650us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.094619 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.094667 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.094750 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2138,"thread_start_us":103,"threads_started":1}
I20260806 11:42:04.095878 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.086292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:43973 (local address 127.0.0.1:40908)
0806 11:42:04.087492 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:04.087510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:04.087615 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:04.090663 (+ 3048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:04.090666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:04.090686 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:04.090776 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.090785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.092100 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.092104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.094320 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.094333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.095515 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.095518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.095610 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.095631 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 11:42:04.095705 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 11:42:04.095762 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":119,"threads_started":1}
I20260806 11:42:04.096009 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.086431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.0.113.3:43973)
0806 11:42:04.090157 (+ 3726us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.090168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.090187 (+ 19us) server_negotiation.cc:407] Connection header received
0806 11:42:04.090272 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.090279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.090505 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.090676 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:04.090876 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.091953 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.094505 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.095375 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.095513 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.095648 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.095753 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.095855 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3616,"thread_start_us":111,"threads_started":1}
I20260806 11:42:04.095645 10540 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff652e389fc4b8796dcf5cedcf56c90 (DEFAULT_TABLE table=test-table [id=0ae44f81266f428790d7ffdba182a5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:04.096894 10540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff652e389fc4b8796dcf5cedcf56c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:04.102783 10935 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff652e389fc4b8796dcf5cedcf56c90 (DEFAULT_TABLE table=test-table [id=0ae44f81266f428790d7ffdba182a5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:04.103652 10935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff652e389fc4b8796dcf5cedcf56c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:04.103220 10733 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff652e389fc4b8796dcf5cedcf56c90 (DEFAULT_TABLE table=test-table [id=0ae44f81266f428790d7ffdba182a5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:04.104277 10733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff652e389fc4b8796dcf5cedcf56c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:04.105868 11074 tablet_bootstrap.cc:492] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: Bootstrap starting.
I20260806 11:42:04.108448 11074 tablet_bootstrap.cc:654] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:04.109483 11074 log.cc:824] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:04.110459 11076 tablet_bootstrap.cc:492] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: Bootstrap starting.
I20260806 11:42:04.112260 11074 tablet_bootstrap.cc:492] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: No bootstrap required, opened a new log
I20260806 11:42:04.112495 11074 ts_tablet_manager.cc:1397] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260806 11:42:04.112880 11076 tablet_bootstrap.cc:654] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:04.113798 11076 log.cc:824] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:04.115512 11074 raft_consensus.cc:348] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:04.115702 11076 tablet_bootstrap.cc:492] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: No bootstrap required, opened a new log
I20260806 11:42:04.115748 11074 raft_consensus.cc:374] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:04.114559 11077 tablet_bootstrap.cc:492] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: Bootstrap starting.
I20260806 11:42:04.115833 11074 raft_consensus.cc:729] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8f1735feb8499f95df54292701beec, State: Initialized, Role: FOLLOWER
I20260806 11:42:04.115902 11076 ts_tablet_manager.cc:1397] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 11:42:04.116345 11074 consensus_queue.cc:260] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:04.117362 11074 ts_tablet_manager.cc:1428] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260806 11:42:04.117969 10634 heartbeater.cc:503] Master 127.0.113.62:44983 was elected leader, sending a full tablet report...
I20260806 11:42:04.118860 11077 tablet_bootstrap.cc:654] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:04.119745 11077 log.cc:824] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:04.122678 11077 tablet_bootstrap.cc:492] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: No bootstrap required, opened a new log
I20260806 11:42:04.122912 11077 ts_tablet_manager.cc:1397] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260806 11:42:04.123008 11076 raft_consensus.cc:348] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:04.123265 11076 raft_consensus.cc:374] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:04.123324 11076 raft_consensus.cc:729] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11654e1660a64ad7a09ed46610715c1b, State: Initialized, Role: FOLLOWER
I20260806 11:42:04.123749 11076 consensus_queue.cc:260] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:04.124713 11076 ts_tablet_manager.cc:1428] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 11:42:04.126232 11025 heartbeater.cc:503] Master 127.0.113.62:44983 was elected leader, sending a full tablet report...
I20260806 11:42:04.128172 11077 raft_consensus.cc:348] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:04.128458 11077 raft_consensus.cc:374] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:04.128539 11077 raft_consensus.cc:729] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6850047bf5d34a4fa628dd2366429165, State: Initialized, Role: FOLLOWER
I20260806 11:42:04.128940 11077 consensus_queue.cc:260] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:04.129634 10828 heartbeater.cc:503] Master 127.0.113.62:44983 was elected leader, sending a full tablet report...
I20260806 11:42:04.129774 11077 ts_tablet_manager.cc:1428] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 11:42:04.255465 11081 raft_consensus.cc:482] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:42:04.255722 11081 raft_consensus.cc:504] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:04.257026 11081 leader_election.cc:288] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8f1735feb8499f95df54292701beec (127.0.113.1:33483), 6850047bf5d34a4fa628dd2366429165 (127.0.113.2:33957)
I20260806 11:42:04.261457 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.257600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:36555 (local address 127.0.113.2:33957)
0806 11:42:04.257758 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.257766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.257787 (+ 21us) server_negotiation.cc:407] Connection header received
0806 11:42:04.257960 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.257964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.258024 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.258124 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:04.258445 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.259149 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.260251 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.260704 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.260792 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.260884 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.261266 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.261339 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:42:04.261487 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.257457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:33957 (local address 127.0.113.3:36555)
0806 11:42:04.257600 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:04.257620 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:04.257749 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:04.258161 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:04.258164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:04.258190 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:04.258325 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.258336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.259261 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.259267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.260101 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.260111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.260853 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.260856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.260978 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.261003 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 11:42:04.261131 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 11:42:04.261302 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":41}
I20260806 11:42:04.261490 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.257457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:33483 (local address 127.0.113.3:46965)
0806 11:42:04.258034 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:04.258049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:04.258136 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:04.258404 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:04.258409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:04.258429 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:04.258545 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.258552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.259377 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.259382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.260129 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:04.260137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.260885 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.260889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:04.261010 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.261027 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 11:42:04.261132 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 11:42:04.261262 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":459}
I20260806 11:42:04.262600 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.257704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:46965 (local address 127.0.113.1:33483)
0806 11:42:04.257841 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.257847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.258153 (+ 306us) server_negotiation.cc:407] Connection header received
0806 11:42:04.258213 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.258216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.258265 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.258370 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:04.258662 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.259221 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.260337 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.260737 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.260863 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.260924 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.262375 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.262452 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 11:42:04.262661 10766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff652e389fc4b8796dcf5cedcf56c90" candidate_uuid: "11654e1660a64ad7a09ed46610715c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6850047bf5d34a4fa628dd2366429165" is_pre_election: true
I20260806 11:42:04.263007 10766 raft_consensus.cc:2454] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11654e1660a64ad7a09ed46610715c1b in term 0.
I20260806 11:42:04.263232 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff652e389fc4b8796dcf5cedcf56c90" candidate_uuid: "11654e1660a64ad7a09ed46610715c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8f1735feb8499f95df54292701beec" is_pre_election: true
I20260806 11:42:04.263530 10567 raft_consensus.cc:2454] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11654e1660a64ad7a09ed46610715c1b in term 0.
I20260806 11:42:04.263610 10858 leader_election.cc:302] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11654e1660a64ad7a09ed46610715c1b, 6850047bf5d34a4fa628dd2366429165; no voters:
I20260806 11:42:04.264258 11081 raft_consensus.cc:2781] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:42:04.264410 11081 raft_consensus.cc:482] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:42:04.264484 11081 raft_consensus.cc:3037] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:04.266124 11081 raft_consensus.cc:504] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:04.266721 11081 leader_election.cc:288] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [CANDIDATE]: Term 1 election: Requested vote from peers dd8f1735feb8499f95df54292701beec (127.0.113.1:33483), 6850047bf5d34a4fa628dd2366429165 (127.0.113.2:33957)
I20260806 11:42:04.266821 10766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff652e389fc4b8796dcf5cedcf56c90" candidate_uuid: "11654e1660a64ad7a09ed46610715c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6850047bf5d34a4fa628dd2366429165"
I20260806 11:42:04.266849 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff652e389fc4b8796dcf5cedcf56c90" candidate_uuid: "11654e1660a64ad7a09ed46610715c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8f1735feb8499f95df54292701beec"
I20260806 11:42:04.266945 10766 raft_consensus.cc:3037] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:04.266978 10567 raft_consensus.cc:3037] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:04.268114 10567 raft_consensus.cc:2454] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11654e1660a64ad7a09ed46610715c1b in term 1.
I20260806 11:42:04.268396 10766 raft_consensus.cc:2454] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11654e1660a64ad7a09ed46610715c1b in term 1.
I20260806 11:42:04.268440 10857 leader_election.cc:302] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11654e1660a64ad7a09ed46610715c1b, dd8f1735feb8499f95df54292701beec; no voters:
I20260806 11:42:04.268623 11081 raft_consensus.cc:2781] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:04.268890 11081 raft_consensus.cc:686] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 1 LEADER]: Becoming Leader. State: Replica: 11654e1660a64ad7a09ed46610715c1b, State: Running, Role: LEADER
I20260806 11:42:04.269456 11081 consensus_queue.cc:237] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:04.273047 9215 catalog_manager.cc:5495] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 11654e1660a64ad7a09ed46610715c1b (127.0.113.3). New cstate: current_term: 1 leader_uuid: "11654e1660a64ad7a09ed46610715c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } health_report { overall_health: UNKNOWN } } }
W20260806 11:42:04.295487 11028 tablet.cc:2369] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:42:04.314635 10831 tablet.cc:2369] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:42:04.328888 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.325225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.0.113.3:43973)
0806 11:42:04.325372 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.325376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.325393 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:42:04.325486 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.325489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.325565 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.325678 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:42:04.325933 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.326443 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.327347 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.327567 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.327682 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.328005 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:42:04.328486 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:42:04.328621 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.328677 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.328735 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:42:04.334720 10766 raft_consensus.cc:1261] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [term 1 FOLLOWER]: Refusing update from remote peer 11654e1660a64ad7a09ed46610715c1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:04.334720 10567 raft_consensus.cc:1261] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [term 1 FOLLOWER]: Refusing update from remote peer 11654e1660a64ad7a09ed46610715c1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:04.335490 11083 consensus_queue.cc:1035] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:04.335673 11081 consensus_queue.cc:1035] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 11:42:04.343456 10637 tablet.cc:2369] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:42:04.344030 11090 mvcc.cc:205] Tried to move back new op lower bound from 7315523683667775488 to 7315523683410395136. Current Snapshot: MvccSnapshot[applied={T|T < 7315523683667775488}]
I20260806 11:42:04.353572 9215 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:04.353739 9215 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:04.357743 9215 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fac143e703c4cb19d4b5ca844af0183: Sending DeleteTablet for 3 replicas of tablet 2ff652e389fc4b8796dcf5cedcf56c90
I20260806 11:42:04.358460 10540 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff652e389fc4b8796dcf5cedcf56c90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:04 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:38842
I20260806 11:42:04.358685 10935 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff652e389fc4b8796dcf5cedcf56c90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:04 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:40908
I20260806 11:42:04.358726 10733 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff652e389fc4b8796dcf5cedcf56c90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:04 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:50294
I20260806 11:42:04.359623 11103 tablet_replica.cc:330] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: stopping tablet replica
I20260806 11:42:04.359992 11103 raft_consensus.cc:2229] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:42:04.360329 11101 tablet_replica.cc:330] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: stopping tablet replica
I20260806 11:42:04.360535 11103 raft_consensus.cc:2258] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:04.360622 11101 raft_consensus.cc:2229] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:04.360929 11101 raft_consensus.cc:2258] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:04.362263 11103 ts_tablet_manager.cc:1905] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:04.362283 11101 ts_tablet_manager.cc:1905] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:04.362525 11102 tablet_replica.cc:330] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: stopping tablet replica
I20260806 11:42:04.362787 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 9174
I20260806 11:42:04.365475 11101 ts_tablet_manager.cc:1918] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:04.365594 11101 log.cc:1196] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/2ff652e389fc4b8796dcf5cedcf56c90
I20260806 11:42:04.365900 11101 ts_tablet_manager.cc:1939] T 2ff652e389fc4b8796dcf5cedcf56c90 P dd8f1735feb8499f95df54292701beec: Deleting consensus metadata
I20260806 11:42:04.367610 11102 raft_consensus.cc:2229] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:04.367944 11102 raft_consensus.cc:2258] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:04.370438 11103 ts_tablet_manager.cc:1918] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:04.370563 11103 log.cc:1196] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/2ff652e389fc4b8796dcf5cedcf56c90
I20260806 11:42:04.370872 11103 ts_tablet_manager.cc:1939] T 2ff652e389fc4b8796dcf5cedcf56c90 P 11654e1660a64ad7a09ed46610715c1b: Deleting consensus metadata
I20260806 11:42:04.373178 11102 ts_tablet_manager.cc:1905] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:04.375871 11102 ts_tablet_manager.cc:1918] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:04.375995 11102 log.cc:1196] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/2ff652e389fc4b8796dcf5cedcf56c90
I20260806 11:42:04.376354 11102 ts_tablet_manager.cc:1939] T 2ff652e389fc4b8796dcf5cedcf56c90 P 6850047bf5d34a4fa628dd2366429165: Deleting consensus metadata
W20260806 11:42:04.382776 10664 connection.cc:535] server connection from 127.0.0.1:50294 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 11:42:04.382925 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.361362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.0.113.61:39161)
0806 11:42:04.361478 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.361482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.372800 (+ 11318us) server_negotiation.cc:407] Connection header received
0806 11:42:04.372854 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.372857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.372911 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.372998 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.374053 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.374807 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.376128 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.376507 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.376591 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.379173 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.379178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.379181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.379213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.380966 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.381490 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.381494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.381495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.381582 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.381945 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.381948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.381950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.382134 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.382334 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.382633 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.382744 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 11:42:04.383126 10859 connection.cc:535] server connection from 127.0.0.1:40908 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 11:42:04.383392 10476 connection.cc:535] server connection from 127.0.0.1:38842 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 11:42:04.394748 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.361068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.0.113.60:46597)
0806 11:42:04.361181 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.361186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.383997 (+ 22811us) server_negotiation.cc:407] Connection header received
0806 11:42:04.384993 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.384999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.385061 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.385151 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.385840 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.386586 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.387735 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.388117 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.388229 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.389944 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.389948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.389951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.389976 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.392466 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.392947 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.392951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.392953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.393031 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.393373 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.393378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.393380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.393640 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.394336 (+ 696us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.394428 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.394562 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 11:42:04.396854 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.386074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.0.113.61:39161)
0806 11:42:04.386167 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.386171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.387153 (+ 982us) server_negotiation.cc:407] Connection header received
0806 11:42:04.387252 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.387255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.387301 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.387380 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.388242 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.389207 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.390591 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.390976 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.391065 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.392845 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.392849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.392852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.392883 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.394762 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.395588 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.395592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.395593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.395684 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.395995 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.395999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.396001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.396173 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.396304 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.396573 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.396687 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 11:42:04.396899 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:04.386017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.0.113.60:46597)
0806 11:42:04.386128 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:04.386132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:04.387734 (+ 1602us) server_negotiation.cc:407] Connection header received
0806 11:42:04.387796 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:04.387799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:04.387858 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:04.387931 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:04.389032 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.389827 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.391062 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:04.391393 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:04.391463 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:04.393101 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:04.393105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:04.393108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:04.393134 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:04.394954 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.395494 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.395498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.395500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.395599 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:04.395912 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:04.395915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:04.395917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:04.396130 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:04.396229 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:04.396606 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 11:42:04.396730 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":46}
I20260806 11:42:05.348901 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.348293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.1:59943)
0806 11:42:05.348646 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.348757 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":86,"threads_started":1}
I20260806 11:42:05.349020 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.348509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.3:57007)
0806 11:42:05.348800 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.348903 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":68,"threads_started":1}
W20260806 11:42:05.349290 10634 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:44983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
W20260806 11:42:05.349291 11025 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:44983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
I20260806 11:42:05.350423 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.350034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.2:57685)
0806 11:42:05.350260 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.350346 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":59,"threads_started":1}
W20260806 11:42:05.350723 10828 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:44983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
I20260806 11:42:05.858834 11121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2fac143e703c4cb19d4b5ca844af0183)
I20260806 11:42:05.858985 11121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } }
I20260806 11:42:05.859560 11121 leader_election.cc:288] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fac143e703c4cb19d4b5ca844af0183 (127.0.113.62:44983), f78d76dabc0946e7a419656c7bac1a81 (127.0.113.61:39161)
I20260806 11:42:05.860184 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.859741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:60176)
0806 11:42:05.860013 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.860094 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":103,"threads_started":1}
I20260806 11:42:05.860219 11127 raft_consensus.cc:482] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2fac143e703c4cb19d4b5ca844af0183)
I20260806 11:42:05.860440 11127 raft_consensus.cc:504] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } }
W20260806 11:42:05.860564 10357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
I20260806 11:42:05.861423 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.861189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:60178)
0806 11:42:05.861300 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.861357 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 11:42:05.861615 10357 leader_election.cc:334] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2fac143e703c4cb19d4b5ca844af0183 (127.0.113.62:44983): Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
I20260806 11:42:05.862061 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.861626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:60186)
0806 11:42:05.861881 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.861948 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":66,"threads_started":1}
W20260806 11:42:05.862223 10082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
I20260806 11:42:05.862241 11127 leader_election.cc:288] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fac143e703c4cb19d4b5ca844af0183 (127.0.113.62:44983), 55b6cc576f2a45cfbcd7a26633595361 (127.0.113.60:46597)
I20260806 11:42:05.863201 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.862959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:60200)
0806 11:42:05.863081 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.863149 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 11:42:05.863379 10082 leader_election.cc:334] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2fac143e703c4cb19d4b5ca844af0183 (127.0.113.62:44983): Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
I20260806 11:42:05.864686 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.859890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.0.113.61:39161)
0806 11:42:05.860516 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:05.860521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:05.860540 (+ 19us) server_negotiation.cc:407] Connection header received
0806 11:42:05.860595 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:05.860598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:05.860646 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:05.860741 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:05.861142 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.861917 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.863646 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.864065 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.864183 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.864245 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:05.864506 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 11:42:05.864579 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":68,"threads_started":1}
I20260806 11:42:05.864683 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.859779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39161 (local address 127.0.0.1:52460)
0806 11:42:05.860383 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.860397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:05.860512 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:05.860800 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:05.860807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:05.860835 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:05.860959 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.860968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.862040 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.862044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.863513 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.863523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.864194 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.864197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.864371 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.864397 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:42:05.864490 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 11:42:05.864562 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":90,"threads_started":1}
I20260806 11:42:05.865141 10118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55b6cc576f2a45cfbcd7a26633595361" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f78d76dabc0946e7a419656c7bac1a81" is_pre_election: true
I20260806 11:42:05.865300 10118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55b6cc576f2a45cfbcd7a26633595361 in term 1.
I20260806 11:42:05.865727 10355 leader_election.cc:302] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55b6cc576f2a45cfbcd7a26633595361, f78d76dabc0946e7a419656c7bac1a81; no voters: 2fac143e703c4cb19d4b5ca844af0183
I20260806 11:42:05.865959 11121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 11:42:05.866034 11121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2fac143e703c4cb19d4b5ca844af0183)
I20260806 11:42:05.866087 11121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 1 FOLLOWER]: Advancing to term 2
I20260806 11:42:05.866184 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.862068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:46597 (local address 127.0.0.1:57592)
0806 11:42:05.862432 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.862446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:05.862538 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:05.862814 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:05.862818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:05.862839 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:05.862936 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.862941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.863852 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.863856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.864761 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.864782 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.865756 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.865761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.865858 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.865884 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:42:05.865970 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 11:42:05.866049 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":59,"threads_started":1}
I20260806 11:42:05.866220 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.862220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.0.113.60:46597)
0806 11:42:05.862574 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:05.862586 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:05.862600 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:42:05.862644 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:05.862646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:05.862692 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:05.862781 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:05.863068 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.863746 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.864927 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.865634 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.865742 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.865785 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:05.865989 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 11:42:05.866103 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260806 11:42:05.866644 10386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f78d76dabc0946e7a419656c7bac1a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "55b6cc576f2a45cfbcd7a26633595361" is_pre_election: true
I20260806 11:42:05.867259 11121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } }
I20260806 11:42:05.867501 10386 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78d76dabc0946e7a419656c7bac1a81 in current term 2: Already voted for candidate 55b6cc576f2a45cfbcd7a26633595361 in this term.
I20260806 11:42:05.867602 11121 leader_election.cc:288] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [CANDIDATE]: Term 2 election: Requested vote from peers 2fac143e703c4cb19d4b5ca844af0183 (127.0.113.62:44983), f78d76dabc0946e7a419656c7bac1a81 (127.0.113.61:39161)
I20260806 11:42:05.867950 10118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55b6cc576f2a45cfbcd7a26633595361" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f78d76dabc0946e7a419656c7bac1a81"
I20260806 11:42:05.868031 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.867680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:60216)
0806 11:42:05.867939 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.867982 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260806 11:42:05.868057 10118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 1 FOLLOWER]: Advancing to term 2
I20260806 11:42:05.868077 10081 leader_election.cc:302] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f78d76dabc0946e7a419656c7bac1a81; no voters: 2fac143e703c4cb19d4b5ca844af0183, 55b6cc576f2a45cfbcd7a26633595361
I20260806 11:42:05.868701 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.868493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:60228)
0806 11:42:05.868599 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.868646 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 11:42:05.868813 10357 leader_election.cc:334] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2fac143e703c4cb19d4b5ca844af0183 (127.0.113.62:44983): Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
I20260806 11:42:05.869182 10118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55b6cc576f2a45cfbcd7a26633595361 in term 2.
I20260806 11:42:05.869306 11127 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:42:05.869524 10355 leader_election.cc:302] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55b6cc576f2a45cfbcd7a26633595361, f78d76dabc0946e7a419656c7bac1a81; no voters: 2fac143e703c4cb19d4b5ca844af0183
I20260806 11:42:05.869787 11121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 11:42:05.870025 11121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [term 2 LEADER]: Becoming Leader. State: Replica: 55b6cc576f2a45cfbcd7a26633595361, State: Running, Role: LEADER
I20260806 11:42:05.870281 11121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } }
I20260806 11:42:05.871519 11136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b6cc576f2a45cfbcd7a26633595361. Latest consensus state: current_term: 2 leader_uuid: "55b6cc576f2a45cfbcd7a26633595361" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:05.871644 11136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:05.871968 11139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:42:05.874444 11139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:42:05.874845 11139 catalog_manager.cc:1259] Loaded cluster ID: 977f3e83e3b54ec4b71ace2eccabe00f
I20260806 11:42:05.874915 11139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:42:05.875449 11139 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:42:05.875836 11139 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Loaded TSK: 0
I20260806 11:42:05.876585 11139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:42:05.912626 10384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:42:05.915787 10384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:42:05.919086 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.918658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:60240)
0806 11:42:05.918776 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.918993 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 11:42:05.919137 10118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [term 2 FOLLOWER]: Refusing update from remote peer 55b6cc576f2a45cfbcd7a26633595361: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 11:42:05.919649 11136 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 11:42:05.919889 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.919651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.0.1:60254)
0806 11:42:05.919765 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.919821 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 11:42:05.920085 10357 consensus_peers.cc:487] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 -> Peer 2fac143e703c4cb19d4b5ca844af0183 (127.0.113.62:44983): Couldn't send request to peer 2fac143e703c4cb19d4b5ca844af0183. Status: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 11:42:05.921515 11127 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b6cc576f2a45cfbcd7a26633595361. Latest consensus state: current_term: 2 leader_uuid: "55b6cc576f2a45cfbcd7a26633595361" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:05.921657 11127 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:42:05.922756 11136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "55b6cc576f2a45cfbcd7a26633595361" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:05.922885 11136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:05.923383 11127 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "55b6cc576f2a45cfbcd7a26633595361" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2fac143e703c4cb19d4b5ca844af0183" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44983 } } peers { permanent_uuid: "f78d76dabc0946e7a419656c7bac1a81" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39161 } attrs { promote: false } } peers { permanent_uuid: "55b6cc576f2a45cfbcd7a26633595361" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 46597 } attrs { promote: false } } }
I20260806 11:42:05.923517 11127 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d76dabc0946e7a419656c7bac1a81 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:42:05.933490 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.928995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:33483 (local address 127.0.0.1:38856)
0806 11:42:05.929148 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.929163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:05.929250 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:05.929849 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:05.929852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:05.929886 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:05.930056 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.930063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.931040 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.931043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.932090 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.932100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.933084 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.933088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.933180 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.933210 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 11:42:05.933285 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 11:42:05.933369 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":123}
I20260806 11:42:05.933640 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.928664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:43973 (local address 127.0.0.1:40926)
0806 11:42:05.928811 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.928825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:05.928903 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:05.929868 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:05.929874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:05.929892 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:05.930200 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.930205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.931098 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.931100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.932402 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.932413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.933259 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.933262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.933349 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.933374 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 11:42:05.933418 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 11:42:05.933481 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":246,"spinlock_wait_cycles":3456}
I20260806 11:42:05.934036 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.929114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.0.113.1:33483)
0806 11:42:05.929580 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:05.929585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:05.929599 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:42:05.929652 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:05.929656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:05.929710 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:05.929934 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:05.930184 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.930932 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.932230 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.932954 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.933708 (+ 754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.933810 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:05.933846 (+ 36us) server_negotiation.cc:299] Negotiation successful
0806 11:42:05.933919 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":89,"threads_started":1}
I20260806 11:42:05.934197 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.929265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.0.113.3:43973)
0806 11:42:05.929572 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:05.929580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:05.929597 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:42:05.929656 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:05.929659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:05.929710 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:05.930024 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:05.930300 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.930946 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.932534 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.933134 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.933885 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.933948 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:05.933981 (+ 33us) server_negotiation.cc:299] Negotiation successful
0806 11:42:05.934048 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":93,"threads_started":1}
I20260806 11:42:05.934340 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.929303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:33957 (local address 127.0.0.1:50298)
0806 11:42:05.929721 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.929735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:05.929839 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:05.930705 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:05.930708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:05.930727 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:05.930803 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.930809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.931690 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.931693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.932870 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:05.932878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.933963 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.933967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:05.934046 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.934066 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 11:42:05.934132 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 11:42:05.934191 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260806 11:42:05.934578 10540 tablet_service.cc:1467] Processing CreateTablet for tablet f95da1e0b7174eba9e94358d0971efd1 (DEFAULT_TABLE table=test-table [id=adfc749be8a643b8bdc5fc8aff66b1cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:05.934757 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.929769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.0.113.2:33957)
0806 11:42:05.930480 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:05.930484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:05.930498 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:42:05.930547 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:05.930549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:05.930590 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:05.930673 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:05.930944 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.931559 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.932997 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:05.933735 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:05.933843 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:05.933905 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:05.934540 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 11:42:05.934614 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":93,"threads_started":1}
I20260806 11:42:05.934697 10935 tablet_service.cc:1467] Processing CreateTablet for tablet f95da1e0b7174eba9e94358d0971efd1 (DEFAULT_TABLE table=test-table [id=adfc749be8a643b8bdc5fc8aff66b1cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:05.934950 10540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95da1e0b7174eba9e94358d0971efd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:05.935017 10935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95da1e0b7174eba9e94358d0971efd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:05.935228 10733 tablet_service.cc:1467] Processing CreateTablet for tablet f95da1e0b7174eba9e94358d0971efd1 (DEFAULT_TABLE table=test-table [id=adfc749be8a643b8bdc5fc8aff66b1cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:05.935510 10733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95da1e0b7174eba9e94358d0971efd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:05.937158 11157 tablet_bootstrap.cc:492] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: Bootstrap starting.
I20260806 11:42:05.937704 11159 tablet_bootstrap.cc:492] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec: Bootstrap starting.
I20260806 11:42:05.938110 11160 tablet_bootstrap.cc:492] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165: Bootstrap starting.
I20260806 11:42:05.938169 11157 tablet_bootstrap.cc:654] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:05.938659 11159 tablet_bootstrap.cc:654] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:05.939052 11160 tablet_bootstrap.cc:654] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:05.939901 11159 tablet_bootstrap.cc:492] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec: No bootstrap required, opened a new log
I20260806 11:42:05.940029 11159 ts_tablet_manager.cc:1397] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:05.940405 11160 tablet_bootstrap.cc:492] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165: No bootstrap required, opened a new log
I20260806 11:42:05.940514 11160 ts_tablet_manager.cc:1397] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:05.940598 11159 raft_consensus.cc:348] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:05.940724 11159 raft_consensus.cc:374] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:05.940765 11159 raft_consensus.cc:729] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8f1735feb8499f95df54292701beec, State: Initialized, Role: FOLLOWER
I20260806 11:42:05.940917 11157 tablet_bootstrap.cc:492] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: No bootstrap required, opened a new log
I20260806 11:42:05.940898 11159 consensus_queue.cc:260] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:05.941038 11157 ts_tablet_manager.cc:1397] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 11:42:05.941042 11160 raft_consensus.cc:348] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:05.941165 11160 raft_consensus.cc:374] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:05.941210 11160 raft_consensus.cc:729] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6850047bf5d34a4fa628dd2366429165, State: Initialized, Role: FOLLOWER
I20260806 11:42:05.941264 11159 ts_tablet_manager.cc:1428] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:05.941519 11160 consensus_queue.cc:260] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:05.941527 11157 raft_consensus.cc:348] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:05.941692 11157 raft_consensus.cc:374] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:05.941733 11157 raft_consensus.cc:729] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11654e1660a64ad7a09ed46610715c1b, State: Initialized, Role: FOLLOWER
I20260806 11:42:05.941905 11160 ts_tablet_manager.cc:1428] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:05.943727 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.941718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.1:54699)
0806 11:42:05.943549 (+ 1831us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.943618 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1749,"thread_start_us":119,"threads_started":1}
I20260806 11:42:05.941911 11157 consensus_queue.cc:260] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:05.944347 10827 heartbeater.cc:503] Master 127.0.113.60:46597 was elected leader, sending a full tablet report...
I20260806 11:42:05.944547 11157 ts_tablet_manager.cc:1428] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 11:42:05.944924 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.944493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.1:46059)
0806 11:42:05.944808 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.944859 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260806 11:42:05.946113 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.945672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.3:42193)
0806 11:42:05.945938 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.946018 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
I20260806 11:42:05.946460 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.942932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.2:58123)
0806 11:42:05.946292 (+ 3360us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.946356 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3222,"thread_start_us":102,"threads_started":1}
I20260806 11:42:05.946813 11024 heartbeater.cc:503] Master 127.0.113.60:46597 was elected leader, sending a full tablet report...
I20260806 11:42:05.947134 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.946911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.2:35119)
0806 11:42:05.947007 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.947061 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 11:42:05.947381 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.947041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.3:37035)
0806 11:42:05.947253 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.947316 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260806 11:42:05.990820 11174 raft_consensus.cc:482] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:42:05.990988 11174 raft_consensus.cc:504] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:05.991483 11174 leader_election.cc:288] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8f1735feb8499f95df54292701beec (127.0.113.1:33483), 6850047bf5d34a4fa628dd2366429165 (127.0.113.2:33957)
I20260806 11:42:05.991971 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95da1e0b7174eba9e94358d0971efd1" candidate_uuid: "11654e1660a64ad7a09ed46610715c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8f1735feb8499f95df54292701beec" is_pre_election: true
I20260806 11:42:05.992045 10766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95da1e0b7174eba9e94358d0971efd1" candidate_uuid: "11654e1660a64ad7a09ed46610715c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6850047bf5d34a4fa628dd2366429165" is_pre_election: true
I20260806 11:42:05.992121 10567 raft_consensus.cc:2454] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11654e1660a64ad7a09ed46610715c1b in term 0.
I20260806 11:42:05.992272 10766 raft_consensus.cc:2454] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11654e1660a64ad7a09ed46610715c1b in term 0.
I20260806 11:42:05.992573 10857 leader_election.cc:302] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11654e1660a64ad7a09ed46610715c1b, dd8f1735feb8499f95df54292701beec; no voters:
I20260806 11:42:05.992781 11174 raft_consensus.cc:2781] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:42:05.992851 11174 raft_consensus.cc:482] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:42:05.992934 11174 raft_consensus.cc:3037] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:05.994123 11174 raft_consensus.cc:504] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:05.994434 11174 leader_election.cc:288] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [CANDIDATE]: Term 1 election: Requested vote from peers dd8f1735feb8499f95df54292701beec (127.0.113.1:33483), 6850047bf5d34a4fa628dd2366429165 (127.0.113.2:33957)
I20260806 11:42:05.994786 10766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95da1e0b7174eba9e94358d0971efd1" candidate_uuid: "11654e1660a64ad7a09ed46610715c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6850047bf5d34a4fa628dd2366429165"
I20260806 11:42:05.994802 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95da1e0b7174eba9e94358d0971efd1" candidate_uuid: "11654e1660a64ad7a09ed46610715c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8f1735feb8499f95df54292701beec"
I20260806 11:42:05.994935 10766 raft_consensus.cc:3037] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:05.994915 10567 raft_consensus.cc:3037] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:05.995829 10567 raft_consensus.cc:2454] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11654e1660a64ad7a09ed46610715c1b in term 1.
I20260806 11:42:05.996007 10766 raft_consensus.cc:2454] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11654e1660a64ad7a09ed46610715c1b in term 1.
I20260806 11:42:05.996173 10857 leader_election.cc:302] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11654e1660a64ad7a09ed46610715c1b, dd8f1735feb8499f95df54292701beec; no voters:
I20260806 11:42:05.996376 11174 raft_consensus.cc:2781] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:05.996577 11174 raft_consensus.cc:686] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 1 LEADER]: Becoming Leader. State: Replica: 11654e1660a64ad7a09ed46610715c1b, State: Running, Role: LEADER
I20260806 11:42:05.996742 11174 consensus_queue.cc:237] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } }
I20260806 11:42:05.997347 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:05.997085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.3:59001)
0806 11:42:05.997193 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:05.997255 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 11:42:05.998413 10382 catalog_manager.cc:5495] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 11654e1660a64ad7a09ed46610715c1b (127.0.113.3). New cstate: current_term: 1 leader_uuid: "11654e1660a64ad7a09ed46610715c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11654e1660a64ad7a09ed46610715c1b" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 43973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 } health_report { overall_health: UNKNOWN } } }
I20260806 11:42:06.016848 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:06.013688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.0.113.3:43973)
0806 11:42:06.013836 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:06.013841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:06.013856 (+ 15us) server_negotiation.cc:407] Connection header received
0806 11:42:06.013980 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:06.013984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:06.014044 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:06.014120 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:42:06.014379 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:06.014884 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:06.015749 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:06.015973 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:06.016084 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:06.016362 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:42:06.016453 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:42:06.016541 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:06.016658 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 11:42:06.016712 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 11:42:06.019104 10567 raft_consensus.cc:1261] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [term 1 FOLLOWER]: Refusing update from remote peer 11654e1660a64ad7a09ed46610715c1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:06.019104 10766 raft_consensus.cc:1261] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [term 1 FOLLOWER]: Refusing update from remote peer 11654e1660a64ad7a09ed46610715c1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:06.019610 11174 consensus_queue.cc:1035] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8f1735feb8499f95df54292701beec" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:06.019800 11176 consensus_queue.cc:1035] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6850047bf5d34a4fa628dd2366429165" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:06.021365 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:06.021116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.1:35981)
0806 11:42:06.021216 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:06.021279 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 11:42:06.021703 10633 heartbeater.cc:503] Master 127.0.113.60:46597 was elected leader, sending a full tablet report...
I20260806 11:42:06.022080 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:06.021751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.2:46025)
0806 11:42:06.021926 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:06.021984 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 11:42:06.023748 11181 mvcc.cc:205] Tried to move back new op lower bound from 7315523690569912320 to 7315523690483941376. Current Snapshot: MvccSnapshot[applied={T|T < 7315523690569912320}]
I20260806 11:42:06.025372 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:06.025008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.2:58693)
0806 11:42:06.025254 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:06.025309 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260806 11:42:06.025379 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:06.024616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.3:37505)
0806 11:42:06.025228 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:06.025293 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538}
I20260806 11:42:06.027379 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:06.025908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.3:35017)
0806 11:42:06.027242 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:06.027287 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1285,"thread_start_us":109,"threads_started":1}
I20260806 11:42:06.027515 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:06.027219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:44983 (local address 127.0.113.1:47923)
0806 11:42:06.027403 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:06.027456 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260806 11:42:06.028268 11183 mvcc.cc:205] Tried to move back new op lower bound from 7315523690569912320 to 7315523690483941376. Current Snapshot: MvccSnapshot[applied={T|T < 7315523690569912320}]
I20260806 11:42:06.032078 10383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:06.032209 10383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:06.035969 10383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55b6cc576f2a45cfbcd7a26633595361: Sending DeleteTablet for 3 replicas of tablet f95da1e0b7174eba9e94358d0971efd1
I20260806 11:42:06.036664 10540 tablet_service.cc:1514] Processing DeleteTablet for tablet f95da1e0b7174eba9e94358d0971efd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:06 UTC) from {username='oryx', principal='oryx/127.0.113.60@KRBTEST.COM'} at 127.0.0.1:38856
I20260806 11:42:06.036744 10733 tablet_service.cc:1514] Processing DeleteTablet for tablet f95da1e0b7174eba9e94358d0971efd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:06 UTC) from {username='oryx', principal='oryx/127.0.113.60@KRBTEST.COM'} at 127.0.0.1:50298
I20260806 11:42:06.036835 10935 tablet_service.cc:1514] Processing DeleteTablet for tablet f95da1e0b7174eba9e94358d0971efd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:06 UTC) from {username='oryx', principal='oryx/127.0.113.60@KRBTEST.COM'} at 127.0.0.1:40926
I20260806 11:42:06.037006 11195 tablet_replica.cc:330] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec: stopping tablet replica
I20260806 11:42:06.037122 11199 tablet_replica.cc:330] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: stopping tablet replica
I20260806 11:42:06.037125 11195 raft_consensus.cc:2229] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:06.037231 11199 raft_consensus.cc:2229] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:42:06.037253 11195 raft_consensus.cc:2258] T f95da1e0b7174eba9e94358d0971efd1 P dd8f1735feb8499f95df54292701beec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:06.037427 11199 raft_consensus.cc:2258] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:06.037762 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 10321
I20260806 11:42:06.050388 11199 ts_tablet_manager.cc:1905] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:06.053083 11199 ts_tablet_manager.cc:1918] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:06.053172 11199 log.cc:1196] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/f95da1e0b7174eba9e94358d0971efd1
I20260806 11:42:06.053455 11199 ts_tablet_manager.cc:1939] T f95da1e0b7174eba9e94358d0971efd1 P 11654e1660a64ad7a09ed46610715c1b: Deleting consensus metadata
I20260806 11:42:06.054236 10355 catalog_manager.cc:4841] TS 11654e1660a64ad7a09ed46610715c1b (127.0.113.3:43973): tablet f95da1e0b7174eba9e94358d0971efd1 (table test-table [id=adfc749be8a643b8bdc5fc8aff66b1cf]) successfully deleted
I20260806 11:42:06.056206 11198 tablet_replica.cc:330] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165: stopping tablet replica
I20260806 11:42:06.056425 11198 raft_consensus.cc:2229] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:06.056528 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:06.056221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:33483 (local address 127.0.0.1:38868)
0806 11:42:06.056386 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:06.056446 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 11:42:06.056584 11198 raft_consensus.cc:2258] T f95da1e0b7174eba9e94358d0971efd1 P 6850047bf5d34a4fa628dd2366429165 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 11:42:06.056713 10353 catalog_manager.cc:4584] TS dd8f1735feb8499f95df54292701beec (127.0.113.1:33483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f95da1e0b7174eba9e94358d0971efd1: Network error: Client connection negotiation failed: client connection to 127.0.113.1:33483: connect: Connection refused (error 111)
I20260806 11:42:06.056835 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 10645
I20260806 11:42:06.069777 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 10836
I20260806 11:42:06.072670 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:06.069840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:33957 (local address 127.0.0.1:50302)
0806 11:42:06.072512 (+ 2672us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:06.072573 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.2:33957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2387}
I20260806 11:42:06.084533 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 9907
I20260806 11:42:06.096819 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 10051
2026-08-06T11:42:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5629 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:42:06 dist-test-slave-45dx krb5kdc[11202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:42:06 dist-test-slave-45dx krb5kdc[11202](info): set up 2 sockets
Aug 06 11:42:06 dist-test-slave-45dx krb5kdc[11202](info): commencing operation
krb5kdc: starting...
W20260806 11:42:07.333495 452 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:42:07 dist-test-slave-45dx krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:42:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:42:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.62.keytab.
Entry for principal oryx/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.62.keytab.
I20260806 11:42:07.454632 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:43095
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.62.keytab
--principal=oryx/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:43095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:07.570073 11219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:07.570360 11219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:07.570402 11219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:07.574334 11219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:07.574401 11219 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 11:42:07.574419 11219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:07.574435 11219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:07.574451 11219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:07.574465 11219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:07.579656 11219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:43095
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.62.keytab
--principal=oryx/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:43095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:07.580684 11219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:07.581660 11219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:07.590179 11231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:07.590245 11219 server_base.cc:1034] running on GCE node
W20260806 11:42:07.590179 11230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:07.590185 11233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:07.590996 11219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:07.592039 11219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:07.593226 11219 hybrid_clock.cc:648] HybridClock initialized: now 1786016527593184 us; error 113 us; skew 500 ppm
Aug 06 11:42:07 dist-test-slave-45dx krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:07.596024 11219 init.cc:377] Logged in from keytab as oryx/127.0.113.62@KRBTEST.COM (short username oryx)
I20260806 11:42:07.596915 11219 webserver.cc:460] Webserver started at http://127.0.113.62:38861/ using document root <none> and password file <none>
I20260806 11:42:07.597401 11219 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:07.597479 11219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:07.597649 11219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:07.599588 11219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "9ace7ce515f742ab9b402dc03f9a8c8d"
format_stamp: "Formatted at 2026-08-06 11:42:07 on dist-test-slave-45dx"
server_key: "47f1f4b3bf0f972d22167297dde53d91"
server_key_iv: "d3e164d3fff3ac715ecaf73b2d943ad0"
server_key_version: "encryptionkey@0"
I20260806 11:42:07.600122 11219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "9ace7ce515f742ab9b402dc03f9a8c8d"
format_stamp: "Formatted at 2026-08-06 11:42:07 on dist-test-slave-45dx"
server_key: "47f1f4b3bf0f972d22167297dde53d91"
server_key_iv: "d3e164d3fff3ac715ecaf73b2d943ad0"
server_key_version: "encryptionkey@0"
I20260806 11:42:07.603645 11219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 11:42:07.606004 11241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:07.607137 11219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:07.607234 11219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "9ace7ce515f742ab9b402dc03f9a8c8d"
format_stamp: "Formatted at 2026-08-06 11:42:07 on dist-test-slave-45dx"
server_key: "47f1f4b3bf0f972d22167297dde53d91"
server_key_iv: "d3e164d3fff3ac715ecaf73b2d943ad0"
server_key_version: "encryptionkey@0"
I20260806 11:42:07.607312 11219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:07.620788 11219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:07.624050 11219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:07.624205 11219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:07.632385 11219 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:43095
I20260806 11:42:07.632421 11330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:43095 every 8 connection(s)
I20260806 11:42:07.633301 11219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:42:07.636000 11332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:07.641182 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 11219
I20260806 11:42:07.641355 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:42:07.641588 11332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: Bootstrap starting.
I20260806 11:42:07.641676 452 external_mini_cluster.cc:1428] Setting key 6ddbde999525bd07083c58bdf7cf17bb
I20260806 11:42:07.644044 11332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:07.644831 11332 log.cc:824] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:07.646592 11332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: No bootstrap required, opened a new log
Aug 06 11:42:07 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:07.650235 11332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace7ce515f742ab9b402dc03f9a8c8d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43095 } }
I20260806 11:42:07.650523 11332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:07.650595 11332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ace7ce515f742ab9b402dc03f9a8c8d, State: Initialized, Role: FOLLOWER
I20260806 11:42:07.651065 11332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace7ce515f742ab9b402dc03f9a8c8d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43095 } }
I20260806 11:42:07.651249 11332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:42:07.651328 11332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:42:07.651422 11332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:07.652577 11332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace7ce515f742ab9b402dc03f9a8c8d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43095 } }
I20260806 11:42:07.652917 11332 leader_election.cc:302] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ace7ce515f742ab9b402dc03f9a8c8d; no voters:
I20260806 11:42:07.653345 11332 leader_election.cc:288] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:42:07.653424 11340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:07.653638 11340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [term 1 LEADER]: Becoming Leader. State: Replica: 9ace7ce515f742ab9b402dc03f9a8c8d, State: Running, Role: LEADER
I20260806 11:42:07.653925 11340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace7ce515f742ab9b402dc03f9a8c8d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43095 } }
I20260806 11:42:07.654639 11332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:42:07.655964 11346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ace7ce515f742ab9b402dc03f9a8c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace7ce515f742ab9b402dc03f9a8c8d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43095 } } }
I20260806 11:42:07.656080 11346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:07.656512 11348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ace7ce515f742ab9b402dc03f9a8c8d. Latest consensus state: current_term: 1 leader_uuid: "9ace7ce515f742ab9b402dc03f9a8c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace7ce515f742ab9b402dc03f9a8c8d" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43095 } } }
I20260806 11:42:07.656584 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:07.643060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.113.62:43095)
0806 11:42:07.643484 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:07.643498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:07.643532 (+ 34us) server_negotiation.cc:407] Connection header received
0806 11:42:07.644225 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:07.644243 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:07.644609 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:07.644927 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:07.645725 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.646713 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.647567 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.648024 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.648194 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:07.650966 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:07.650971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:07.650986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:07.651015 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:07.653084 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:07.653623 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:07.653632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:07.653639 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:07.653790 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:07.654138 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:07.654142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:07.654145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:07.654644 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:07.654860 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:07.655185 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 11:42:07.655503 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":140,"threads_started":1}
I20260806 11:42:07.656608 11348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d [sys.catalog]: This master's current role is: LEADER
W20260806 11:42:07.659584 11368 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:42:07.659672 11368 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:42:07.659776 11360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:42:07.660691 11360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:42:07.666258 11360 catalog_manager.cc:1347] Generated new cluster ID: 4f016569b4db49c2b2a0a0c48751da75
I20260806 11:42:07.666325 11360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:42:07.681644 11360 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:42:07.682605 11360 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:42:07.687676 11360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: Generated new TSK 0
I20260806 11:42:07.688400 11360 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab.
Entry for principal oryx/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab.
I20260806 11:42:07.728770 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:43095
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab
--principal=oryx/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:07.838940 11376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:07.839236 11376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:07.839313 11376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:07.843325 11376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:07.843421 11376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:07.843490 11376 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 11:42:07.843575 11376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:42:07.848744 11376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab
--principal=oryx/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.113.62:43095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:07.849797 11376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:07.850821 11376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:07.860332 11385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:07.860353 11383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:07.860369 11382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:07.860532 11376 server_base.cc:1034] running on GCE node
I20260806 11:42:07.861045 11376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:07.861650 11376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:07.862859 11376 hybrid_clock.cc:648] HybridClock initialized: now 1786016527862851 us; error 31 us; skew 500 ppm
Aug 06 11:42:07 dist-test-slave-45dx krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:07.865495 11376 init.cc:377] Logged in from keytab as oryx/127.0.113.1@KRBTEST.COM (short username oryx)
I20260806 11:42:07.866338 11376 webserver.cc:460] Webserver started at http://127.0.113.1:45199/ using document root <none> and password file <none>
I20260806 11:42:07.866878 11376 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:07.866967 11376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:07.867188 11376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:07.869153 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "81423e6a25074bde8f9ff36fe40433bd"
format_stamp: "Formatted at 2026-08-06 11:42:07 on dist-test-slave-45dx"
server_key: "0b4eb29919e2e40dc66f3066832182c6"
server_key_iv: "806a913ae4d5a93ec29ff39893c6598f"
server_key_version: "encryptionkey@0"
I20260806 11:42:07.869721 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "81423e6a25074bde8f9ff36fe40433bd"
format_stamp: "Formatted at 2026-08-06 11:42:07 on dist-test-slave-45dx"
server_key: "0b4eb29919e2e40dc66f3066832182c6"
server_key_iv: "806a913ae4d5a93ec29ff39893c6598f"
server_key_version: "encryptionkey@0"
I20260806 11:42:07.873160 11376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:42:07.875334 11395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:07.876518 11376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:07.876641 11376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "81423e6a25074bde8f9ff36fe40433bd"
format_stamp: "Formatted at 2026-08-06 11:42:07 on dist-test-slave-45dx"
server_key: "0b4eb29919e2e40dc66f3066832182c6"
server_key_iv: "806a913ae4d5a93ec29ff39893c6598f"
server_key_version: "encryptionkey@0"
I20260806 11:42:07.876744 11376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:07.892429 11376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:07.895192 11376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:07.895385 11376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:07.896935 11376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:07.896999 11376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:07.897050 11376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:07.897075 11376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 11:42:07 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:07.908392 11376 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:46609
I20260806 11:42:07.908444 11564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:46609 every 8 connection(s)
I20260806 11:42:07.909432 11376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:42:07.910905 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:07.897481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:52873 (local address 127.0.113.62:43095)
0806 11:42:07.897610 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:07.897614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:07.898633 (+ 1019us) server_negotiation.cc:407] Connection header received
0806 11:42:07.899604 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:07.899608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:07.899672 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:07.899779 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:07.901535 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.902456 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.903704 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.903940 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.904058 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:07.906997 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:07.907000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:07.907003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:07.907031 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:07.908700 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:07.909267 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:07.909269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:07.909270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:07.909345 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:07.909753 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:07.909755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:07.909756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:07.910031 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:07.910125 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:07.910652 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 11:42:07.910750 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:42:07.911634 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:07.897937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43095 (local address 127.0.113.1:52873)
0806 11:42:07.898478 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:07.898519 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:07.899380 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:07.900066 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:07.900079 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:07.900636 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:07.901334 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:07.901378 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.902636 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.902639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:07.903579 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:07.903588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.904111 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.904116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:07.904280 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:07.905127 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:07.905146 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:07.906795 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:07.908809 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:07.908814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:07.908827 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:07.909164 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:07.909439 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:07.909444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:07.909449 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:07.909646 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:07.910199 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:07.910204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:07.910437 (+ 233us) client_negotiation.cc:769] Sending connection context
0806 11:42:07.910585 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 11:42:07.910882 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":136,"threads_started":1}
I20260806 11:42:07.912948 11565 heartbeater.cc:347] Connected to a master server at 127.0.113.62:43095
I20260806 11:42:07.913100 11565 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:07.913568 11565 heartbeater.cc:511] Master 127.0.113.62:43095 requested a full tablet report, sending...
I20260806 11:42:07.915130 11262 ts_manager.cc:194] Registered new tserver with Master: 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1:46609)
I20260806 11:42:07.915293 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 11376
I20260806 11:42:07.915427 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:42:07.915802 452 external_mini_cluster.cc:1428] Setting key 216498b333c8ce27ec451a4ca90ba8ec
I20260806 11:42:07.916651 11262 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:52873
I20260806 11:42:07.923135 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:07.915006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:56881 (local address 127.0.113.62:43095)
0806 11:42:07.915127 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:07.915132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:07.915400 (+ 268us) server_negotiation.cc:407] Connection header received
0806 11:42:07.915566 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:07.915569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:07.915607 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:07.915686 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:07.916201 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.916920 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.917821 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.918202 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.918322 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:07.919365 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:07.919370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:07.919376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:07.919408 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:07.921218 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:07.921723 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:07.921726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:07.921727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:07.921817 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:07.922154 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:07.922157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:07.922158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:07.922312 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:07.922587 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:07.922900 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 11:42:07.923010 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 11:42:07.923406 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:07.914908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43095 (local address 127.0.113.1:56881)
0806 11:42:07.915290 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:07.915306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:07.915442 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:07.915719 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:07.915724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:07.915929 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:07.916068 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:07.916075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.917065 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.917071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:07.917688 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:07.917697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:07.918321 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:07.918323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:07.918396 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:07.918897 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:07.918908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:07.919200 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:07.921342 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:07.921348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:07.921353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:07.921623 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:07.921913 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:07.921918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:07.921923 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:07.922044 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:07.922582 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:07.922587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:07.922710 (+ 123us) client_negotiation.cc:769] Sending connection context
0806 11:42:07.922827 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 11:42:07.922987 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for oryx/127.0.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab.
Entry for principal oryx/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab.
I20260806 11:42:07.944931 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:43095
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab
--principal=oryx/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:08.052764 11575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:08.053033 11575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:08.053076 11575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:08.057374 11575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:08.057433 11575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:08.057472 11575 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 11:42:08.057548 11575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:42:08.062759 11575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab
--principal=oryx/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.113.62:43095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:08.063786 11575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:08.064782 11575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 11:42:08.073978 11575 server_base.cc:1034] running on GCE node
W20260806 11:42:08.073923 11582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:08.073923 11583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:08.073949 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:08.074565 11575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:08.075111 11575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:08.076351 11575 hybrid_clock.cc:648] HybridClock initialized: now 1786016528076321 us; error 38 us; skew 500 ppm
Aug 06 11:42:08 dist-test-slave-45dx krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:08.079010 11575 init.cc:377] Logged in from keytab as oryx/127.0.113.2@KRBTEST.COM (short username oryx)
I20260806 11:42:08.079924 11575 webserver.cc:460] Webserver started at http://127.0.113.2:33259/ using document root <none> and password file <none>
I20260806 11:42:08.080508 11575 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:08.080572 11575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:08.080722 11575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:08.082592 11575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "f8ee3845af8f4483a006c82d4fa5b11d"
format_stamp: "Formatted at 2026-08-06 11:42:08 on dist-test-slave-45dx"
server_key: "006bca52ba1698399841fdf7a3828f82"
server_key_iv: "4209f806c0a403d1a03007ece87f009a"
server_key_version: "encryptionkey@0"
I20260806 11:42:08.083106 11575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "f8ee3845af8f4483a006c82d4fa5b11d"
format_stamp: "Formatted at 2026-08-06 11:42:08 on dist-test-slave-45dx"
server_key: "006bca52ba1698399841fdf7a3828f82"
server_key_iv: "4209f806c0a403d1a03007ece87f009a"
server_key_version: "encryptionkey@0"
I20260806 11:42:08.086406 11575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:42:08.088599 11595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:08.089762 11575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:08.089849 11575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "f8ee3845af8f4483a006c82d4fa5b11d"
format_stamp: "Formatted at 2026-08-06 11:42:08 on dist-test-slave-45dx"
server_key: "006bca52ba1698399841fdf7a3828f82"
server_key_iv: "4209f806c0a403d1a03007ece87f009a"
server_key_version: "encryptionkey@0"
I20260806 11:42:08.089921 11575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:08.109503 11575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:08.112537 11575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:08.112679 11575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:08.114113 11575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:08.114210 11575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:08.114307 11575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:08.114346 11575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 11:42:08 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:08.125670 11575 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:33617
I20260806 11:42:08.125991 11768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:33617 every 8 connection(s)
I20260806 11:42:08.126683 11575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:42:08.128674 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.115000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:42855 (local address 127.0.113.62:43095)
0806 11:42:08.115181 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.115188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.116010 (+ 822us) server_negotiation.cc:407] Connection header received
0806 11:42:08.117108 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.117112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.117174 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.117230 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:08.118763 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.119360 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.120630 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.120979 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.121101 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.124814 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:08.124818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:08.124821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:08.124846 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:08.126564 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.127186 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.127189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.127191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.127277 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.127698 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.127700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.127701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.127879 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:08.127969 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.128457 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.128554 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 11:42:08.129472 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.115371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43095 (local address 127.0.113.2:42855)
0806 11:42:08.115845 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.115893 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.116853 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.117716 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.117727 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.118078 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:08.118614 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.118628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.119502 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.119505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.120505 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.120515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.121141 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.121146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.121358 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.122028 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:08.122048 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:08.124641 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:08.126685 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.126692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.126708 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.127051 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.127413 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.127416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.127418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.127580 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.127965 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:08.127970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:08.128169 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.128384 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.128638 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":139,"threads_started":1}
I20260806 11:42:08.130740 11769 heartbeater.cc:347] Connected to a master server at 127.0.113.62:43095
I20260806 11:42:08.130898 11769 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:08.131139 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 11575
I20260806 11:42:08.131268 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:42:08.131521 11769 heartbeater.cc:511] Master 127.0.113.62:43095 requested a full tablet report, sending...
I20260806 11:42:08.131615 452 external_mini_cluster.cc:1428] Setting key 2a41e078903cb213b26bd7dd89a8a5a8
I20260806 11:42:08.132586 11263 ts_manager.cc:194] Registered new tserver with Master: f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617)
I20260806 11:42:08.133271 11263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:42855
I20260806 11:42:08.140830 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.133200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43095 (local address 127.0.113.2:43113)
0806 11:42:08.133558 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.133570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.133682 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.133919 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.133925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.134150 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:08.134299 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.134309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.135260 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.135263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.135802 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.135809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.136421 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.136424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.136491 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.136983 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:08.136995 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:08.137276 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:08.139271 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.139274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.139275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.139481 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.139800 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.139802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.139804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.139881 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.140440 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:08.140442 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:08.140516 (+ 74us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.140599 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.140709 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":105,"threads_started":1}
I20260806 11:42:08.140995 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.133239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:43113 (local address 127.0.113.62:43095)
0806 11:42:08.133351 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.133355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.133642 (+ 287us) server_negotiation.cc:407] Connection header received
0806 11:42:08.133771 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.133775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.133819 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.133897 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:08.134435 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.135122 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.135916 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.136265 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.136426 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.137413 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:08.137419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:08.137424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:08.137454 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:08.139148 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.139591 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.139597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.139601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.139701 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.139975 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.139980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.139984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.140342 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:08.140460 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.140701 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.140819 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for oryx/127.0.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab.
Entry for principal oryx/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab.
I20260806 11:42:08.162429 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:43095
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab
--principal=oryx/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:08.277673 11784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:08.278008 11784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:08.278054 11784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:08.282172 11784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:08.282229 11784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:08.282269 11784 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 11:42:08.282300 11784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:42:08.287389 11784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab
--principal=oryx/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.113.62:43095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:08.288424 11784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:08.289407 11784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:08.299007 11790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:08.299139 11789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:08.298995 11793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:08.299417 11784 server_base.cc:1034] running on GCE node
I20260806 11:42:08.299832 11784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:08.300556 11784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:08.301743 11784 hybrid_clock.cc:648] HybridClock initialized: now 1786016528301723 us; error 29 us; skew 500 ppm
Aug 06 11:42:08 dist-test-slave-45dx krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:08.304493 11784 init.cc:377] Logged in from keytab as oryx/127.0.113.3@KRBTEST.COM (short username oryx)
I20260806 11:42:08.305354 11784 webserver.cc:460] Webserver started at http://127.0.113.3:32837/ using document root <none> and password file <none>
I20260806 11:42:08.305909 11784 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:08.306021 11784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:08.306265 11784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:08.308235 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "65a1763226cf462aa521fdb39d5f9f0d"
format_stamp: "Formatted at 2026-08-06 11:42:08 on dist-test-slave-45dx"
server_key: "9c86507ed4df49403009767e28295154"
server_key_iv: "3cee82b9785ec3b64583ca7a15465a10"
server_key_version: "encryptionkey@0"
I20260806 11:42:08.308873 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "65a1763226cf462aa521fdb39d5f9f0d"
format_stamp: "Formatted at 2026-08-06 11:42:08 on dist-test-slave-45dx"
server_key: "9c86507ed4df49403009767e28295154"
server_key_iv: "3cee82b9785ec3b64583ca7a15465a10"
server_key_version: "encryptionkey@0"
I20260806 11:42:08.312486 11784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:42:08.314769 11807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:08.315877 11784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:08.316012 11784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "65a1763226cf462aa521fdb39d5f9f0d"
format_stamp: "Formatted at 2026-08-06 11:42:08 on dist-test-slave-45dx"
server_key: "9c86507ed4df49403009767e28295154"
server_key_iv: "3cee82b9785ec3b64583ca7a15465a10"
server_key_version: "encryptionkey@0"
I20260806 11:42:08.316120 11784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:08.328241 11784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:08.330983 11784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:08.331223 11784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:08.332830 11784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:08.332903 11784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:08.332971 11784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:08.333025 11784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 11:42:08 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:08.344141 11784 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:41757
I20260806 11:42:08.344205 11969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:41757 every 8 connection(s)
I20260806 11:42:08.345187 11784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:42:08.346481 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.333451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:49551 (local address 127.0.113.62:43095)
0806 11:42:08.333608 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.333614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.334407 (+ 793us) server_negotiation.cc:407] Connection header received
0806 11:42:08.335333 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.335340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.335404 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.335508 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:08.336734 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.337320 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.338587 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.339005 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.339140 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.342546 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:08.342549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:08.342552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:08.342605 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:08.344378 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.344857 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.344863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.344867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.344971 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.345319 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.345323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.345325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.345574 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:08.345690 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.346245 (+ 555us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.346355 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:42:08.347193 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.333799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43095 (local address 127.0.113.3:49551)
0806 11:42:08.334242 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.334290 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.335128 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.335684 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.335693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.336065 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:08.336579 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.336592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.337454 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.337458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.338429 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.338439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.339189 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.339195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.339403 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.340052 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:08.340078 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:08.342310 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:08.344486 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.344491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.344500 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.344750 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.345065 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.345067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.345069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.345195 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.345694 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:08.345700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:08.345964 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.346149 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.346396 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":130,"threads_started":1}
I20260806 11:42:08.348451 11970 heartbeater.cc:347] Connected to a master server at 127.0.113.62:43095
I20260806 11:42:08.348601 11970 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:08.348918 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 11784
I20260806 11:42:08.349066 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:42:08.349187 11970 heartbeater.cc:511] Master 127.0.113.62:43095 requested a full tablet report, sending...
I20260806 11:42:08.349411 452 external_mini_cluster.cc:1428] Setting key b6ac7a54fef5636a1a235c5402037b7e
I20260806 11:42:08.350363 11262 ts_manager.cc:194] Registered new tserver with Master: 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757)
I20260806 11:42:08.351043 11262 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:49551
I20260806 11:42:08.351533 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:42:08.359061 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.350626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43095 (local address 127.0.113.3:36627)
0806 11:42:08.351403 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.351418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.351514 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.351764 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.351766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.351990 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:08.352125 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.352134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.353107 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.353110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.353639 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.353646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.354348 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.354354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.354476 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.355059 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:08.355078 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:08.355511 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:08.357595 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.357598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.357599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.357780 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.358122 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.358126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.358127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.358234 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.358688 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:08.358690 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:08.358761 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.358839 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.358939 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":89,"threads_started":1}
I20260806 11:42:08.359238 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.350627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:36627 (local address 127.0.113.62:43095)
0806 11:42:08.350734 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.350740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.351497 (+ 757us) server_negotiation.cc:407] Connection header received
0806 11:42:08.351605 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.351608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.351651 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.351732 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:08.352259 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.352988 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.353754 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.354118 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.354220 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.355674 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:08.355677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:08.355679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:08.355701 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:08.357486 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.357889 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.357895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.357900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.357999 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.358343 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.358349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.358353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.358588 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:08.358698 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.358938 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.359049 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 11:42:08 dist-test-slave-45dx krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016528, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 11:42:08 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:08 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 11:42:08.378382 11337 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:08.372516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.113.62:43095)
0806 11:42:08.372627 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.372633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.372649 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:42:08.372680 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.372682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.372713 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.372780 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:08.373482 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.373963 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.374727 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.375006 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.375098 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.378263 (+ 3165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47506: BlockingRecv error: recv got EOF from 127.0.0.1:47506 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 11:42:08 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016528, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:08.387475 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.379928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.0.113.62:43095)
0806 11:42:08.380038 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.380042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.380186 (+ 144us) server_negotiation.cc:407] Connection header received
0806 11:42:08.380275 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.380277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.380354 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.380435 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:08.381055 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.381508 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.382252 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.382451 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.382523 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.384666 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:08.384669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:08.384671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:08.384692 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:08.386176 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.386521 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.386523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.386525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.386595 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.386825 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.386827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.386828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.386978 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:08.387075 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.387244 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.387335 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 11:42:08.390373 11262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:42:08.393465 11262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:42:08.408032 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.402428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:46609 (local address 127.0.0.1:49208)
0806 11:42:08.403296 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.403319 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.403409 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.404176 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.404179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.404195 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:08.404296 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.404335 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.405645 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.405651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.406689 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.406699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.407464 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.407467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.407579 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.407620 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.407754 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.407875 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":67,"thread_start_us":100,"threads_started":1}
I20260806 11:42:08.408166 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.402001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41757 (local address 127.0.0.1:47308)
0806 11:42:08.402727 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.402761 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.402842 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.403606 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.403612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.403651 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:08.403874 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.403880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.405255 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.405258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.406726 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.406738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.407766 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.407769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.407847 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.407867 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.407927 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.407981 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":108,"spinlock_wait_cycles":14336,"thread_start_us":105,"threads_started":1}
I20260806 11:42:08.408437 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.401728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:33617 (local address 127.0.0.1:41448)
0806 11:42:08.402376 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.402422 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.402545 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.403610 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.403613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.403653 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:08.403894 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.403903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.405063 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.405069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.406726 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.406738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.407972 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.407976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.408072 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.408093 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.408166 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.408236 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":217,"spinlock_wait_cycles":4096,"thread_start_us":98,"threads_started":1}
I20260806 11:42:08.408670 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.402710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.0.113.1:46609)
0806 11:42:08.403575 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.403592 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.403610 (+ 18us) server_negotiation.cc:407] Connection header received
0806 11:42:08.403679 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.403685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.403882 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.404018 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:08.404441 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.405504 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.406812 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.407333 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.407462 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.407639 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.408405 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.408492 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":93,"threads_started":1}
I20260806 11:42:08.408818 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.402170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.0.113.3:41757)
0806 11:42:08.403121 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.403134 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.403150 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:42:08.403236 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.403244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.403442 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.403615 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:08.403992 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.405125 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.406872 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.407645 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.407751 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.407902 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.407958 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.408049 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":140,"threads_started":1}
I20260806 11:42:08.408891 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.402389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.0.113.2:33617)
0806 11:42:08.402948 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.402962 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.402988 (+ 26us) server_negotiation.cc:407] Connection header received
0806 11:42:08.403085 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.403091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.403291 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.403447 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:08.404031 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.404953 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.406871 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.407645 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.407749 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.407902 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.408633 (+ 731us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.408727 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":112,"threads_started":1}
I20260806 11:42:08.411588 11671 tablet_service.cc:1467] Processing CreateTablet for tablet c1d9cd646df94e6fb05a169f4672e41b (DEFAULT_TABLE table=test-table [id=1dce4dc051804b9cb2777718f4afa617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:08.411592 11462 tablet_service.cc:1467] Processing CreateTablet for tablet c1d9cd646df94e6fb05a169f4672e41b (DEFAULT_TABLE table=test-table [id=1dce4dc051804b9cb2777718f4afa617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:08.411585 11878 tablet_service.cc:1467] Processing CreateTablet for tablet c1d9cd646df94e6fb05a169f4672e41b (DEFAULT_TABLE table=test-table [id=1dce4dc051804b9cb2777718f4afa617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:08.412632 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d9cd646df94e6fb05a169f4672e41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:08.412632 11462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d9cd646df94e6fb05a169f4672e41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:08.412673 11671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d9cd646df94e6fb05a169f4672e41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:08.419216 12002 tablet_bootstrap.cc:492] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: Bootstrap starting.
I20260806 11:42:08.420342 12005 tablet_bootstrap.cc:492] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: Bootstrap starting.
I20260806 11:42:08.420933 12003 tablet_bootstrap.cc:492] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: Bootstrap starting.
I20260806 11:42:08.422446 12002 tablet_bootstrap.cc:654] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:08.422569 12005 tablet_bootstrap.cc:654] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:08.423221 12002 log.cc:824] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:08.423312 12003 tablet_bootstrap.cc:654] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:08.423372 12005 log.cc:824] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:08.424052 12003 log.cc:824] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:08.425221 12002 tablet_bootstrap.cc:492] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: No bootstrap required, opened a new log
I20260806 11:42:08.425282 12005 tablet_bootstrap.cc:492] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: No bootstrap required, opened a new log
I20260806 11:42:08.425433 12002 ts_tablet_manager.cc:1397] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 11:42:08.425479 12005 ts_tablet_manager.cc:1397] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:42:08.425882 12003 tablet_bootstrap.cc:492] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: No bootstrap required, opened a new log
I20260806 11:42:08.426049 12003 ts_tablet_manager.cc:1397] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:42:08.428895 12002 raft_consensus.cc:348] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } }
I20260806 11:42:08.429123 12002 raft_consensus.cc:374] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:08.429177 12002 raft_consensus.cc:729] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ee3845af8f4483a006c82d4fa5b11d, State: Initialized, Role: FOLLOWER
I20260806 11:42:08.429364 12005 raft_consensus.cc:348] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } }
I20260806 11:42:08.429622 12005 raft_consensus.cc:374] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:08.429677 12005 raft_consensus.cc:729] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a1763226cf462aa521fdb39d5f9f0d, State: Initialized, Role: FOLLOWER
I20260806 11:42:08.429651 12002 consensus_queue.cc:260] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } }
I20260806 11:42:08.429708 12003 raft_consensus.cc:348] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } }
I20260806 11:42:08.429939 12003 raft_consensus.cc:374] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:08.429996 12003 raft_consensus.cc:729] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81423e6a25074bde8f9ff36fe40433bd, State: Initialized, Role: FOLLOWER
I20260806 11:42:08.430083 12005 consensus_queue.cc:260] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } }
I20260806 11:42:08.430442 11769 heartbeater.cc:503] Master 127.0.113.62:43095 was elected leader, sending a full tablet report...
I20260806 11:42:08.430426 12003 consensus_queue.cc:260] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } }
I20260806 11:42:08.430742 12002 ts_tablet_manager.cc:1428] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 11:42:08.431336 11970 heartbeater.cc:503] Master 127.0.113.62:43095 was elected leader, sending a full tablet report...
I20260806 11:42:08.431468 11565 heartbeater.cc:503] Master 127.0.113.62:43095 was elected leader, sending a full tablet report...
I20260806 11:42:08.431711 12005 ts_tablet_manager.cc:1428] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 11:42:08.432261 12003 ts_tablet_manager.cc:1428] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260806 11:42:08.597980 11971 tablet.cc:2369] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:42:08.628348 11771 tablet.cc:2369] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:42:08.641255 12013 raft_consensus.cc:482] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:42:08.641474 12013 raft_consensus.cc:504] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } }
I20260806 11:42:08.642745 12013 leader_election.cc:288] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617), 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757)
I20260806 11:42:08.647387 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.643093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:43563 (local address 127.0.113.2:33617)
0806 11:42:08.643224 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.643229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.643690 (+ 461us) server_negotiation.cc:407] Connection header received
0806 11:42:08.643820 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.643823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.643878 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.643964 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:08.644557 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.645307 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.646376 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.646780 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.646912 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.646978 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.647172 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.647258 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 11:42:08.647419 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.643233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:44301 (local address 127.0.113.3:41757)
0806 11:42:08.643361 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.643365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.643505 (+ 140us) server_negotiation.cc:407] Connection header received
0806 11:42:08.644009 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.644013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.644062 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.644137 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:08.644471 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.645307 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.646380 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.646780 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.646948 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.647036 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.647208 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.647282 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:42:08.647428 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.642979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:33617 (local address 127.0.113.1:43563)
0806 11:42:08.643583 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.643596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.643708 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.644243 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.644246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.644262 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:08.644443 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.644449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.645423 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.645425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.646239 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.646248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.646915 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.646919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.647022 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.647049 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.647136 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.647214 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":335,"thread_start_us":131,"threads_started":1}
I20260806 11:42:08.647434 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.642979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41757 (local address 127.0.113.1:44301)
0806 11:42:08.643390 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.643409 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.643866 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.644158 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.644160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.644175 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:08.644295 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.644350 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.645478 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.645481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.646250 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.646258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.646951 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.646954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.647078 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.647104 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.647193 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.647262 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":5,"thread_start_us":127,"threads_started":1}
I20260806 11:42:08.648370 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d9cd646df94e6fb05a169f4672e41b" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" is_pre_election: true
I20260806 11:42:08.648379 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d9cd646df94e6fb05a169f4672e41b" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a1763226cf462aa521fdb39d5f9f0d" is_pre_election: true
I20260806 11:42:08.648684 11901 raft_consensus.cc:2454] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 0.
I20260806 11:42:08.648684 11697 raft_consensus.cc:2454] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 0.
I20260806 11:42:08.649160 11400 leader_election.cc:302] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81423e6a25074bde8f9ff36fe40433bd, f8ee3845af8f4483a006c82d4fa5b11d; no voters:
I20260806 11:42:08.649467 12013 raft_consensus.cc:2781] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:42:08.649585 12013 raft_consensus.cc:482] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:42:08.649652 12013 raft_consensus.cc:3037] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:08.650732 12013 raft_consensus.cc:504] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } }
I20260806 11:42:08.651119 12013 leader_election.cc:288] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 election: Requested vote from peers f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617), 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757)
I20260806 11:42:08.651589 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d9cd646df94e6fb05a169f4672e41b" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a1763226cf462aa521fdb39d5f9f0d"
I20260806 11:42:08.651589 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d9cd646df94e6fb05a169f4672e41b" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d"
I20260806 11:42:08.651731 11697 raft_consensus.cc:3037] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:08.651731 11901 raft_consensus.cc:3037] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:08.653213 11697 raft_consensus.cc:2454] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 1.
I20260806 11:42:08.653546 11400 leader_election.cc:302] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81423e6a25074bde8f9ff36fe40433bd, f8ee3845af8f4483a006c82d4fa5b11d; no voters:
I20260806 11:42:08.653589 11901 raft_consensus.cc:2454] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 1.
I20260806 11:42:08.653755 12013 raft_consensus.cc:2781] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:08.653992 12013 raft_consensus.cc:686] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 1 LEADER]: Becoming Leader. State: Replica: 81423e6a25074bde8f9ff36fe40433bd, State: Running, Role: LEADER
I20260806 11:42:08.654354 12013 consensus_queue.cc:237] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } }
I20260806 11:42:08.657766 11261 catalog_manager.cc:5495] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1). New cstate: current_term: 1 leader_uuid: "81423e6a25074bde8f9ff36fe40433bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } health_report { overall_health: HEALTHY } } }
W20260806 11:42:08.666661 11566 tablet.cc:2369] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:42:08.702744 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.699141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.0.113.1:46609)
0806 11:42:08.699276 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.699280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.699295 (+ 15us) server_negotiation.cc:407] Connection header received
0806 11:42:08.699437 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.699439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.699497 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.699585 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:42:08.699805 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.700289 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.701195 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.701449 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.701529 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.701874 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:42:08.702299 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:42:08.702396 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.702520 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.702571 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:42:08.708498 11901 raft_consensus.cc:1261] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Refusing update from remote peer 81423e6a25074bde8f9ff36fe40433bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:08.708498 11697 raft_consensus.cc:1261] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Refusing update from remote peer 81423e6a25074bde8f9ff36fe40433bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:08.709172 12020 consensus_queue.cc:1035] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:08.709383 12013 consensus_queue.cc:1035] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:08.718585 12028 mvcc.cc:205] Tried to move back new op lower bound from 7315523701582864384 to 7315523701371346944. Current Snapshot: MvccSnapshot[applied={T|T < 7315523701582864384}]
I20260806 11:42:08.724761 11263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:08.724951 11263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:08.727434 11263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: Sending DeleteTablet for 3 replicas of tablet c1d9cd646df94e6fb05a169f4672e41b
I20260806 11:42:08.728240 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet c1d9cd646df94e6fb05a169f4672e41b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:08 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:47308
I20260806 11:42:08.728478 11671 tablet_service.cc:1514] Processing DeleteTablet for tablet c1d9cd646df94e6fb05a169f4672e41b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:08 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:41448
I20260806 11:42:08.728964 11462 tablet_service.cc:1514] Processing DeleteTablet for tablet c1d9cd646df94e6fb05a169f4672e41b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:08 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:49208
I20260806 11:42:08.728920 11263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:42:08.729354 11263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:42:08.729543 12042 tablet_replica.cc:330] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: stopping tablet replica
I20260806 11:42:08.729867 12042 raft_consensus.cc:2229] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:42:08.730216 12042 raft_consensus.cc:2258] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:08.730574 12040 tablet_replica.cc:330] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: stopping tablet replica
I20260806 11:42:08.730922 12040 raft_consensus.cc:2229] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:08.731248 12040 raft_consensus.cc:2258] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:08.732035 12042 ts_tablet_manager.cc:1905] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:08.732762 12040 ts_tablet_manager.cc:1905] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:08.734679 11462 tablet_service.cc:1467] Processing CreateTablet for tablet b15c8990e5b84105801ea0256ece8c9b (DEFAULT_TABLE table=test-table [id=a0694f9ff70a4ee0b8e02df8647ed9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:08.734982 11462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15c8990e5b84105801ea0256ece8c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:08.734997 12042 ts_tablet_manager.cc:1918] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:08.735102 12042 log.cc:1196] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/c1d9cd646df94e6fb05a169f4672e41b
I20260806 11:42:08.735082 11878 tablet_service.cc:1467] Processing CreateTablet for tablet b15c8990e5b84105801ea0256ece8c9b (DEFAULT_TABLE table=test-table [id=a0694f9ff70a4ee0b8e02df8647ed9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:08.735327 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15c8990e5b84105801ea0256ece8c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:08.735443 12042 ts_tablet_manager.cc:1939] T c1d9cd646df94e6fb05a169f4672e41b P 81423e6a25074bde8f9ff36fe40433bd: Deleting consensus metadata
I20260806 11:42:08.736594 11243 catalog_manager.cc:4841] TS 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1:46609): tablet c1d9cd646df94e6fb05a169f4672e41b (table test-table [id=1dce4dc051804b9cb2777718f4afa617]) successfully deleted
I20260806 11:42:08.736937 12041 tablet_replica.cc:330] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: stopping tablet replica
I20260806 11:42:08.737226 12041 raft_consensus.cc:2229] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:08.737470 11671 tablet_service.cc:1467] Processing CreateTablet for tablet b15c8990e5b84105801ea0256ece8c9b (DEFAULT_TABLE table=test-table [id=a0694f9ff70a4ee0b8e02df8647ed9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:08.737526 12041 raft_consensus.cc:2258] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:08.737723 11671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15c8990e5b84105801ea0256ece8c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:08.739506 12041 ts_tablet_manager.cc:1905] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:08.740209 12002 tablet_bootstrap.cc:492] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: Bootstrap starting.
I20260806 11:42:08.740274 12003 tablet_bootstrap.cc:492] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: Bootstrap starting.
I20260806 11:42:08.741194 12002 tablet_bootstrap.cc:654] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:08.741317 12003 tablet_bootstrap.cc:654] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:08.741954 12005 tablet_bootstrap.cc:492] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: Bootstrap starting.
I20260806 11:42:08.742015 12041 ts_tablet_manager.cc:1918] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:08.742094 12041 log.cc:1196] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/c1d9cd646df94e6fb05a169f4672e41b
I20260806 11:42:08.742841 12041 ts_tablet_manager.cc:1939] T c1d9cd646df94e6fb05a169f4672e41b P f8ee3845af8f4483a006c82d4fa5b11d: Deleting consensus metadata
I20260806 11:42:08.743063 12005 tablet_bootstrap.cc:654] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:08.744086 11243 catalog_manager.cc:4841] TS f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617): tablet c1d9cd646df94e6fb05a169f4672e41b (table test-table [id=1dce4dc051804b9cb2777718f4afa617]) successfully deleted
I20260806 11:42:08.744946 12003 tablet_bootstrap.cc:492] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: No bootstrap required, opened a new log
I20260806 11:42:08.745036 12003 ts_tablet_manager.cc:1397] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 11:42:08.745193 12005 tablet_bootstrap.cc:492] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: No bootstrap required, opened a new log
I20260806 11:42:08.745283 12005 ts_tablet_manager.cc:1397] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 11:42:08.745587 12003 raft_consensus.cc:348] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.745714 12003 raft_consensus.cc:374] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:08.745756 12003 raft_consensus.cc:729] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81423e6a25074bde8f9ff36fe40433bd, State: Initialized, Role: FOLLOWER
I20260806 11:42:08.745788 12005 raft_consensus.cc:348] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.745903 12005 raft_consensus.cc:374] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:08.745944 12005 raft_consensus.cc:729] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a1763226cf462aa521fdb39d5f9f0d, State: Initialized, Role: FOLLOWER
I20260806 11:42:08.745895 12003 consensus_queue.cc:260] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.746102 12005 consensus_queue.cc:260] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.746171 12003 ts_tablet_manager.cc:1428] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:08.747023 12002 tablet_bootstrap.cc:492] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: No bootstrap required, opened a new log
I20260806 11:42:08.747125 12002 ts_tablet_manager.cc:1397] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 11:42:08.747588 12002 raft_consensus.cc:348] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.747692 12002 raft_consensus.cc:374] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:08.747726 12002 raft_consensus.cc:729] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ee3845af8f4483a006c82d4fa5b11d, State: Initialized, Role: FOLLOWER
I20260806 11:42:08.747859 12002 consensus_queue.cc:260] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.748133 12002 ts_tablet_manager.cc:1428] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:08.749419 12005 ts_tablet_manager.cc:1428] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260806 11:42:08.745271 12040 ts_tablet_manager.cc:1918] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:08.749763 12040 log.cc:1196] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/c1d9cd646df94e6fb05a169f4672e41b
I20260806 11:42:08.750254 12040 ts_tablet_manager.cc:1939] T c1d9cd646df94e6fb05a169f4672e41b P 65a1763226cf462aa521fdb39d5f9f0d: Deleting consensus metadata
I20260806 11:42:08.751549 11244 catalog_manager.cc:4841] TS 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757): tablet c1d9cd646df94e6fb05a169f4672e41b (table test-table [id=1dce4dc051804b9cb2777718f4afa617]) successfully deleted
I20260806 11:42:08.880551 12031 raft_consensus.cc:482] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:42:08.880708 12031 raft_consensus.cc:504] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.881155 12031 leader_election.cc:288] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617), 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757)
I20260806 11:42:08.881703 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15c8990e5b84105801ea0256ece8c9b" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" is_pre_election: true
I20260806 11:42:08.881711 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15c8990e5b84105801ea0256ece8c9b" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a1763226cf462aa521fdb39d5f9f0d" is_pre_election: true
I20260806 11:42:08.881903 11901 raft_consensus.cc:2454] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 0.
I20260806 11:42:08.881906 11697 raft_consensus.cc:2454] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 0.
I20260806 11:42:08.882206 11401 leader_election.cc:302] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a1763226cf462aa521fdb39d5f9f0d, 81423e6a25074bde8f9ff36fe40433bd; no voters:
I20260806 11:42:08.882402 12031 raft_consensus.cc:2781] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:42:08.882491 12031 raft_consensus.cc:482] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:42:08.882547 12031 raft_consensus.cc:3037] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:08.883558 12031 raft_consensus.cc:504] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.883929 12031 leader_election.cc:288] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 election: Requested vote from peers f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617), 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757)
I20260806 11:42:08.884341 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15c8990e5b84105801ea0256ece8c9b" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a1763226cf462aa521fdb39d5f9f0d"
I20260806 11:42:08.884385 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15c8990e5b84105801ea0256ece8c9b" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d"
I20260806 11:42:08.884481 11697 raft_consensus.cc:3037] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:08.884497 11901 raft_consensus.cc:3037] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:08.885411 11697 raft_consensus.cc:2454] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 1.
I20260806 11:42:08.885449 11901 raft_consensus.cc:2454] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 1.
I20260806 11:42:08.885704 11400 leader_election.cc:302] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81423e6a25074bde8f9ff36fe40433bd, f8ee3845af8f4483a006c82d4fa5b11d; no voters:
I20260806 11:42:08.885912 12031 raft_consensus.cc:2781] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:08.886029 12031 raft_consensus.cc:686] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 1 LEADER]: Becoming Leader. State: Replica: 81423e6a25074bde8f9ff36fe40433bd, State: Running, Role: LEADER
I20260806 11:42:08.886180 12031 consensus_queue.cc:237] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.887745 11261 catalog_manager.cc:5495] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1). New cstate: current_term: 1 leader_uuid: "81423e6a25074bde8f9ff36fe40433bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } health_report { overall_health: UNKNOWN } } }
Aug 06 11:42:08 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.62@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:08.927114 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.918017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43095 (local address 127.0.0.1:47526)
0806 11:42:08.918279 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.918295 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.918408 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.918684 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.918687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.919155 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:08.919256 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.919263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.920111 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.920115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.921041 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.921053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.921507 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.921510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.921598 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.922189 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:08.922203 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:08.923837 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:08.925696 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.925704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.925709 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.925942 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.926209 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:08.926211 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:08.926212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:08.926288 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:08.926625 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:08.926630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:08.926725 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.926813 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.926943 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260806 11:42:08.927134 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.918095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.0.113.62:43095)
0806 11:42:08.918344 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.918351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.918367 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:42:08.918522 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.918525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.918571 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.918666 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:08.919404 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.919938 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.921168 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.921393 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.921474 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.923982 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:08.923985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:08.923987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:08.924008 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:08.925592 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.926043 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.926045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.926047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.926128 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:08.926366 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:08.926369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:08.926370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:08.926536 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:08.926613 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.926877 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.927004 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260806 11:42:08.930955 11261 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:47526:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 11:42:08.931701 11261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:42:08.936278 11462 tablet_service.cc:1467] Processing CreateTablet for tablet 61c5a6a5aa5a4e3eac1d75a15ea5058b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16f2f2956407495d90cbd7e5bb18423c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 11:42:08.936443 11671 tablet_service.cc:1467] Processing CreateTablet for tablet 61c5a6a5aa5a4e3eac1d75a15ea5058b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16f2f2956407495d90cbd7e5bb18423c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 11:42:08.936489 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 61c5a6a5aa5a4e3eac1d75a15ea5058b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16f2f2956407495d90cbd7e5bb18423c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 11:42:08.936720 11462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c5a6a5aa5a4e3eac1d75a15ea5058b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:08.936841 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c5a6a5aa5a4e3eac1d75a15ea5058b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:08.936858 11671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c5a6a5aa5a4e3eac1d75a15ea5058b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:08.939226 12002 tablet_bootstrap.cc:492] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: Bootstrap starting.
I20260806 11:42:08.939503 12003 tablet_bootstrap.cc:492] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: Bootstrap starting.
I20260806 11:42:08.939848 12005 tablet_bootstrap.cc:492] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: Bootstrap starting.
I20260806 11:42:08.940297 12003 tablet_bootstrap.cc:654] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:08.940289 12002 tablet_bootstrap.cc:654] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:08.940783 12005 tablet_bootstrap.cc:654] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:08.941740 12002 tablet_bootstrap.cc:492] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: No bootstrap required, opened a new log
I20260806 11:42:08.941844 12002 ts_tablet_manager.cc:1397] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 11:42:08.941942 12005 tablet_bootstrap.cc:492] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: No bootstrap required, opened a new log
I20260806 11:42:08.942003 12003 tablet_bootstrap.cc:492] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: No bootstrap required, opened a new log
I20260806 11:42:08.942018 12005 ts_tablet_manager.cc:1397] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:08.942085 12003 ts_tablet_manager.cc:1397] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 11:42:08.942333 12002 raft_consensus.cc:348] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.942459 12002 raft_consensus.cc:374] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:08.942504 12002 raft_consensus.cc:729] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ee3845af8f4483a006c82d4fa5b11d, State: Initialized, Role: FOLLOWER
I20260806 11:42:08.942456 12005 raft_consensus.cc:348] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.942456 12003 raft_consensus.cc:348] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.942564 12003 raft_consensus.cc:374] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:08.942564 12005 raft_consensus.cc:374] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:08.942608 12005 raft_consensus.cc:729] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a1763226cf462aa521fdb39d5f9f0d, State: Initialized, Role: FOLLOWER
I20260806 11:42:08.942608 12003 raft_consensus.cc:729] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81423e6a25074bde8f9ff36fe40433bd, State: Initialized, Role: FOLLOWER
I20260806 11:42:08.942641 12002 consensus_queue.cc:260] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.942725 12003 consensus_queue.cc:260] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.942725 12005 consensus_queue.cc:260] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.942934 12002 ts_tablet_manager.cc:1428] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 11:42:08.942983 12005 ts_tablet_manager.cc:1428] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:08.942983 12003 ts_tablet_manager.cc:1428] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 11:42:08.943001 12012 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:08.943125 12012 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:42:08.943169 12031 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:08.943297 12031 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:42:08.943230 12014 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:08.943349 12014 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:42:08.960536 12014 raft_consensus.cc:482] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:42:08.960708 12014 raft_consensus.cc:504] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.961817 12014 leader_election.cc:288] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1:46609), f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617)
I20260806 11:42:08.967110 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.962135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:37619 (local address 127.0.113.1:46609)
0806 11:42:08.962295 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.962302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.962640 (+ 338us) server_negotiation.cc:407] Connection header received
0806 11:42:08.962706 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.962709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.962766 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.962853 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:08.963169 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.963968 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.965802 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.966412 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.966519 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.966585 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.966893 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.966969 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 11:42:08.967110 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.962566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:37347 (local address 127.0.113.2:33617)
0806 11:42:08.962716 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.962722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.962764 (+ 42us) server_negotiation.cc:407] Connection header received
0806 11:42:08.962979 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.962986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.963027 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.963096 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:08.963353 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.964068 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.965802 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.966404 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.966519 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.966583 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.966893 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.966966 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 11:42:08.967195 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.962440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:33617 (local address 127.0.113.3:37347)
0806 11:42:08.962695 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.962708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.962813 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.963132 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.963136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.963156 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:08.963255 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.963259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.964167 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.964170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.965648 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.965661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.966535 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.966539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.966689 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.966778 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.966855 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.966957 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":212,"thread_start_us":71,"threads_started":1}
I20260806 11:42:08.967193 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.962016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:46609 (local address 127.0.113.3:37619)
0806 11:42:08.962303 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.962321 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.962459 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.962884 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.962887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.962905 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:08.963036 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.963040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.964076 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.964078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.965648 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.965661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.966535 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.966539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.966717 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.966781 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.966855 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.966963 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":188,"thread_start_us":119,"threads_started":1}
I20260806 11:42:08.967756 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c5a6a5aa5a4e3eac1d75a15ea5058b" candidate_uuid: "65a1763226cf462aa521fdb39d5f9f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" is_pre_election: true
I20260806 11:42:08.967816 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c5a6a5aa5a4e3eac1d75a15ea5058b" candidate_uuid: "65a1763226cf462aa521fdb39d5f9f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81423e6a25074bde8f9ff36fe40433bd" is_pre_election: true
I20260806 11:42:08.967947 11697 raft_consensus.cc:2454] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a1763226cf462aa521fdb39d5f9f0d in term 0.
I20260806 11:42:08.968076 11493 raft_consensus.cc:2454] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a1763226cf462aa521fdb39d5f9f0d in term 0.
I20260806 11:42:08.968382 11810 leader_election.cc:302] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a1763226cf462aa521fdb39d5f9f0d, f8ee3845af8f4483a006c82d4fa5b11d; no voters:
I20260806 11:42:08.968640 12014 raft_consensus.cc:2781] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:42:08.968719 12014 raft_consensus.cc:482] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:42:08.968809 12014 raft_consensus.cc:3037] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:08.970054 12014 raft_consensus.cc:504] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.970407 12014 leader_election.cc:288] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [CANDIDATE]: Term 1 election: Requested vote from peers 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1:46609), f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617)
I20260806 11:42:08.970852 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c5a6a5aa5a4e3eac1d75a15ea5058b" candidate_uuid: "65a1763226cf462aa521fdb39d5f9f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d"
I20260806 11:42:08.970799 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c5a6a5aa5a4e3eac1d75a15ea5058b" candidate_uuid: "65a1763226cf462aa521fdb39d5f9f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81423e6a25074bde8f9ff36fe40433bd"
I20260806 11:42:08.970996 11697 raft_consensus.cc:3037] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:08.971050 11493 raft_consensus.cc:3037] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:08.971884 11697 raft_consensus.cc:2454] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a1763226cf462aa521fdb39d5f9f0d in term 1.
I20260806 11:42:08.972054 11493 raft_consensus.cc:2454] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a1763226cf462aa521fdb39d5f9f0d in term 1.
I20260806 11:42:08.972230 11810 leader_election.cc:302] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a1763226cf462aa521fdb39d5f9f0d, f8ee3845af8f4483a006c82d4fa5b11d; no voters:
I20260806 11:42:08.972461 12014 raft_consensus.cc:2781] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:08.972679 12014 raft_consensus.cc:686] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 LEADER]: Becoming Leader. State: Replica: 65a1763226cf462aa521fdb39d5f9f0d, State: Running, Role: LEADER
I20260806 11:42:08.972929 12014 consensus_queue.cc:237] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:08.974149 12065 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: TxnStatusTablet state changed. Reason: New leader 65a1763226cf462aa521fdb39d5f9f0d. Latest consensus state: current_term: 1 leader_uuid: "65a1763226cf462aa521fdb39d5f9f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:08.974260 12065 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: This TxnStatusTablet replica's current role is: LEADER
I20260806 11:42:08.974550 12067 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 11:42:08.974704 12067 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 11:42:08.975298 11260 catalog_manager.cc:5495] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3). New cstate: current_term: 1 leader_uuid: "65a1763226cf462aa521fdb39d5f9f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } health_report { overall_health: HEALTHY } } }
I20260806 11:42:08.987679 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.983686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.0.113.3:41757)
0806 11:42:08.984053 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:08.984057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:08.984071 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:42:08.984126 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:08.984128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:08.984174 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:08.984255 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:42:08.984574 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.985065 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.986285 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.986505 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.986580 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.986916 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:42:08.987291 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:42:08.987390 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:08.987520 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 11:42:08.987571 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260806 11:42:08.987732 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:08.983615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41757 (local address 127.0.0.1:47324)
0806 11:42:08.983841 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:08.983856 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:08.983997 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:08.984329 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:08.984332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:08.984342 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:42:08.984453 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.984460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.985175 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.985178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.986151 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:08.986162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:08.986615 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:08.986619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:08.986716 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:08.986770 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:42:08.987405 (+ 635us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:42:08.987421 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 11:42:08.987502 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 11:42:08.987590 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260806 11:42:08.990856 11697 raft_consensus.cc:1261] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Refusing update from remote peer 65a1763226cf462aa521fdb39d5f9f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:08.990921 11493 raft_consensus.cc:1261] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Refusing update from remote peer 65a1763226cf462aa521fdb39d5f9f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:08.991456 12014 consensus_queue.cc:1035] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:08.991705 12065 consensus_queue.cc:1035] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:08.993575 12031 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: TxnStatusTablet state changed. Reason: New leader 65a1763226cf462aa521fdb39d5f9f0d. Latest consensus state: current_term: 1 leader_uuid: "65a1763226cf462aa521fdb39d5f9f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:08.993564 12012 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: TxnStatusTablet state changed. Reason: New leader 65a1763226cf462aa521fdb39d5f9f0d. Latest consensus state: current_term: 1 leader_uuid: "65a1763226cf462aa521fdb39d5f9f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:08.993697 12031 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:42:08.993698 12012 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:42:08.997040 12012 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65a1763226cf462aa521fdb39d5f9f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:08.997148 12012 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P f8ee3845af8f4483a006c82d4fa5b11d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:42:08.997676 12014 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65a1763226cf462aa521fdb39d5f9f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:08.997785 12014 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: This TxnStatusTablet replica's current role is: LEADER
I20260806 11:42:09.008555 12065 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65a1763226cf462aa521fdb39d5f9f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:09.008692 12065 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 65a1763226cf462aa521fdb39d5f9f0d: This TxnStatusTablet replica's current role is: LEADER
I20260806 11:42:09.010115 12025 mvcc.cc:205] Tried to move back new op lower bound from 7315523702740643840 to 7315523702675365888. Current Snapshot: MvccSnapshot[applied={T|T < 7315523702740643840}]
I20260806 11:42:09.010315 12031 tablet_replica.cc:439] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65a1763226cf462aa521fdb39d5f9f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } } }
I20260806 11:42:09.010445 12031 tablet_replica.cc:442] T 61c5a6a5aa5a4e3eac1d75a15ea5058b P 81423e6a25074bde8f9ff36fe40433bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 11:42:09.011446 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:09.007198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:46017 (local address 127.0.113.3:41757)
0806 11:42:09.007323 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:09.007328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:09.007536 (+ 208us) server_negotiation.cc:407] Connection header received
0806 11:42:09.007665 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:09.007669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:09.007744 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:09.007833 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:42:09.008261 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.008962 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.010216 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.010567 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.010680 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:09.010995 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:42:09.011073 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:42:09.011145 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:09.011272 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 11:42:09.011322 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:42:09.011514 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:09.007094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41757 (local address 127.0.113.1:46017)
0806 11:42:09.007436 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:09.007451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:09.007558 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:09.008004 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:09.008007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:09.008017 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:42:09.008122 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:09.008128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.009076 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.009080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:09.010076 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:09.010088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.010697 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.010700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:09.010790 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:09.010841 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:42:09.011173 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:42:09.011179 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 11:42:09.011252 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 11:42:09.011342 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
I20260806 11:42:09.018317 11697 raft_consensus.cc:1261] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Refusing update from remote peer 81423e6a25074bde8f9ff36fe40433bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:09.018323 11901 raft_consensus.cc:1261] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Refusing update from remote peer 81423e6a25074bde8f9ff36fe40433bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:09.018728 12031 consensus_queue.cc:1035] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:09.018858 12013 consensus_queue.cc:1035] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:09.026863 12027 mvcc.cc:205] Tried to move back new op lower bound from 7315523702855270400 to 7315523702318776320. Current Snapshot: MvccSnapshot[applied={T|T < 7315523702855270400}]
I20260806 11:42:09.042935 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:09.038877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:51645 (local address 127.0.113.1:46609)
0806 11:42:09.038995 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:09.039000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:09.039013 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:42:09.039169 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:09.039173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:09.039228 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:09.039280 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:42:09.039701 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.040529 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.041498 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.041855 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.041967 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:09.042462 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:42:09.042547 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:42:09.042637 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:09.042737 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 11:42:09.042790 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:42:09.042966 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:09.038565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:46609 (local address 127.0.113.3:51645)
0806 11:42:09.038900 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:09.038918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:09.039038 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:09.039435 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:09.039440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:09.039449 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 11:42:09.039581 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:09.039587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.040633 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.040636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:09.041355 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:09.041365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.042050 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.042053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:09.042152 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:09.042204 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 11:42:09.042645 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 11:42:09.042655 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 11:42:09.042718 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 11:42:09.042798 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":128,"threads_started":1}
I20260806 11:42:09.063190 11263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:09.063371 11263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:09.065515 11263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: Sending DeleteTablet for 3 replicas of tablet b15c8990e5b84105801ea0256ece8c9b
I20260806 11:42:09.066160 11671 tablet_service.cc:1514] Processing DeleteTablet for tablet b15c8990e5b84105801ea0256ece8c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:09 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:41448
I20260806 11:42:09.066323 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet b15c8990e5b84105801ea0256ece8c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:09 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:47308
I20260806 11:42:09.066380 11462 tablet_service.cc:1514] Processing DeleteTablet for tablet b15c8990e5b84105801ea0256ece8c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:09 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:49208
I20260806 11:42:09.066392 12041 tablet_replica.cc:330] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: stopping tablet replica
I20260806 11:42:09.066504 12040 tablet_replica.cc:330] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: stopping tablet replica
I20260806 11:42:09.066558 12041 raft_consensus.cc:2229] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:09.066622 12040 raft_consensus.cc:2229] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:09.066684 12041 raft_consensus.cc:2258] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:09.066735 12040 raft_consensus.cc:2258] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:09.066850 12042 tablet_replica.cc:330] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: stopping tablet replica
I20260806 11:42:09.066983 12042 raft_consensus.cc:2229] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:42:09.067184 12042 raft_consensus.cc:2258] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:09.067539 12040 ts_tablet_manager.cc:1905] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:09.068146 12042 ts_tablet_manager.cc:1905] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:09.069001 12041 ts_tablet_manager.cc:1905] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:09.071707 12041 ts_tablet_manager.cc:1918] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:42:09.071777 12041 log.cc:1196] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/b15c8990e5b84105801ea0256ece8c9b
I20260806 11:42:09.072044 12041 ts_tablet_manager.cc:1939] T b15c8990e5b84105801ea0256ece8c9b P f8ee3845af8f4483a006c82d4fa5b11d: Deleting consensus metadata
I20260806 11:42:09.072045 12042 ts_tablet_manager.cc:1918] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:42:09.072248 12042 log.cc:1196] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/b15c8990e5b84105801ea0256ece8c9b
I20260806 11:42:09.072615 12042 ts_tablet_manager.cc:1939] T b15c8990e5b84105801ea0256ece8c9b P 81423e6a25074bde8f9ff36fe40433bd: Deleting consensus metadata
I20260806 11:42:09.073464 11243 catalog_manager.cc:4841] TS 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1:46609): tablet b15c8990e5b84105801ea0256ece8c9b (table test-table [id=a0694f9ff70a4ee0b8e02df8647ed9cb]) successfully deleted
I20260806 11:42:09.074380 12040 ts_tablet_manager.cc:1918] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:42:09.074463 12040 log.cc:1196] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/b15c8990e5b84105801ea0256ece8c9b
I20260806 11:42:09.074707 12040 ts_tablet_manager.cc:1939] T b15c8990e5b84105801ea0256ece8c9b P 65a1763226cf462aa521fdb39d5f9f0d: Deleting consensus metadata
I20260806 11:42:09.075417 11244 catalog_manager.cc:4841] TS 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757): tablet b15c8990e5b84105801ea0256ece8c9b (table test-table [id=a0694f9ff70a4ee0b8e02df8647ed9cb]) successfully deleted
I20260806 11:42:09.075780 11243 catalog_manager.cc:4841] TS f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617): tablet b15c8990e5b84105801ea0256ece8c9b (table test-table [id=a0694f9ff70a4ee0b8e02df8647ed9cb]) successfully deleted
Aug 06 11:42:09 dist-test-slave-45dx krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 11:42:09 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:09 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 11:42:09.096506 12057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:09.089109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.0.113.62:43095)
0806 11:42:09.089273 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:09.089278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:09.089444 (+ 166us) server_negotiation.cc:407] Connection header received
0806 11:42:09.089602 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:09.089605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:09.089649 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:09.089711 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:09.090608 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.091151 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.092348 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.092691 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.092772 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:09.096305 (+ 3533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47532: BlockingRecv error: recv got EOF from 127.0.0.1:47532 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 11:42:09 dist-test-slave-45dx krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM
I20260806 11:42:09.106686 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:09.098180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.0.113.62:43095)
0806 11:42:09.098302 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:09.098306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:09.098507 (+ 201us) server_negotiation.cc:407] Connection header received
0806 11:42:09.098595 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:09.098598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:09.098640 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:09.098713 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:09.099264 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.099762 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.100610 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.100836 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.100941 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:09.103414 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:09.103418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:09.103421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:09.103460 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:09.104980 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:09.105431 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:09.105436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:09.105438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:09.105532 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:09.105817 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:09.105821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:09.105823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:09.106003 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:09.106147 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:09.106390 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 11:42:09.106523 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:42:09.109373 11263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:42:09.109858 11263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:42:09.114974 11462 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5860c55ebb432f8ab1f8583e05fd63 (DEFAULT_TABLE table=test-table [id=adf9e28e3e164beba8980622fb15dc42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:09.115131 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5860c55ebb432f8ab1f8583e05fd63 (DEFAULT_TABLE table=test-table [id=adf9e28e3e164beba8980622fb15dc42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:09.115316 11462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5860c55ebb432f8ab1f8583e05fd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:09.115413 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5860c55ebb432f8ab1f8583e05fd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:09.117678 11671 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5860c55ebb432f8ab1f8583e05fd63 (DEFAULT_TABLE table=test-table [id=adf9e28e3e164beba8980622fb15dc42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:09.117987 11671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5860c55ebb432f8ab1f8583e05fd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:09.117993 12005 tablet_bootstrap.cc:492] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: Bootstrap starting.
I20260806 11:42:09.119002 12005 tablet_bootstrap.cc:654] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:09.120234 12003 tablet_bootstrap.cc:492] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: Bootstrap starting.
I20260806 11:42:09.120499 12005 tablet_bootstrap.cc:492] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: No bootstrap required, opened a new log
I20260806 11:42:09.120580 12005 ts_tablet_manager.cc:1397] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 11:42:09.121054 12003 tablet_bootstrap.cc:654] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:09.121064 12005 raft_consensus.cc:348] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.121196 12005 raft_consensus.cc:374] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:09.121229 12005 raft_consensus.cc:729] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a1763226cf462aa521fdb39d5f9f0d, State: Initialized, Role: FOLLOWER
I20260806 11:42:09.121349 12005 consensus_queue.cc:260] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.121548 12002 tablet_bootstrap.cc:492] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: Bootstrap starting.
I20260806 11:42:09.122021 12005 ts_tablet_manager.cc:1428] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:09.122476 12002 tablet_bootstrap.cc:654] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:09.123943 12003 tablet_bootstrap.cc:492] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: No bootstrap required, opened a new log
I20260806 11:42:09.124040 12003 ts_tablet_manager.cc:1397] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260806 11:42:09.124485 12003 raft_consensus.cc:348] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.124577 12003 raft_consensus.cc:374] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:09.124599 12003 raft_consensus.cc:729] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81423e6a25074bde8f9ff36fe40433bd, State: Initialized, Role: FOLLOWER
I20260806 11:42:09.124729 12003 consensus_queue.cc:260] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.125018 12003 ts_tablet_manager.cc:1428] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:09.126135 12002 tablet_bootstrap.cc:492] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: No bootstrap required, opened a new log
I20260806 11:42:09.126219 12002 ts_tablet_manager.cc:1397] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 11:42:09.126621 12002 raft_consensus.cc:348] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.126731 12002 raft_consensus.cc:374] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:09.126760 12002 raft_consensus.cc:729] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ee3845af8f4483a006c82d4fa5b11d, State: Initialized, Role: FOLLOWER
I20260806 11:42:09.126871 12002 consensus_queue.cc:260] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.127110 12002 ts_tablet_manager.cc:1428] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:42:09.130038 12031 raft_consensus.cc:482] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:42:09.130133 12031 raft_consensus.cc:504] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.130525 12031 leader_election.cc:288] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617), 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757)
I20260806 11:42:09.130885 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5860c55ebb432f8ab1f8583e05fd63" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a1763226cf462aa521fdb39d5f9f0d" is_pre_election: true
I20260806 11:42:09.131037 11901 raft_consensus.cc:2454] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 0.
I20260806 11:42:09.131381 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5860c55ebb432f8ab1f8583e05fd63" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" is_pre_election: true
I20260806 11:42:09.131527 11697 raft_consensus.cc:2454] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 0.
I20260806 11:42:09.131914 11400 leader_election.cc:302] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81423e6a25074bde8f9ff36fe40433bd, f8ee3845af8f4483a006c82d4fa5b11d; no voters:
I20260806 11:42:09.132115 12031 raft_consensus.cc:2781] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:42:09.132187 12031 raft_consensus.cc:482] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:42:09.132239 12031 raft_consensus.cc:3037] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:09.133353 12031 raft_consensus.cc:504] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.133720 12031 leader_election.cc:288] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 election: Requested vote from peers f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617), 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757)
I20260806 11:42:09.134032 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5860c55ebb432f8ab1f8583e05fd63" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a1763226cf462aa521fdb39d5f9f0d"
I20260806 11:42:09.134042 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5860c55ebb432f8ab1f8583e05fd63" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d"
I20260806 11:42:09.134131 11901 raft_consensus.cc:3037] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:09.134151 11697 raft_consensus.cc:3037] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:09.135241 11901 raft_consensus.cc:2454] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 1.
I20260806 11:42:09.135320 11697 raft_consensus.cc:2454] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 1.
I20260806 11:42:09.135558 11401 leader_election.cc:302] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a1763226cf462aa521fdb39d5f9f0d, 81423e6a25074bde8f9ff36fe40433bd; no voters:
I20260806 11:42:09.135708 12031 raft_consensus.cc:2781] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:09.135795 12031 raft_consensus.cc:686] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 1 LEADER]: Becoming Leader. State: Replica: 81423e6a25074bde8f9ff36fe40433bd, State: Running, Role: LEADER
I20260806 11:42:09.135927 12031 consensus_queue.cc:237] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.137413 11263 catalog_manager.cc:5495] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1). New cstate: current_term: 1 leader_uuid: "81423e6a25074bde8f9ff36fe40433bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } health_report { overall_health: UNKNOWN } } }
I20260806 11:42:09.153276 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:09.149271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.0.113.1:46609)
0806 11:42:09.149401 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:09.149405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:09.149534 (+ 129us) server_negotiation.cc:407] Connection header received
0806 11:42:09.149655 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:09.149658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:09.149716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:09.149807 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:42:09.150078 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.150817 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.151999 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:09.152390 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:09.152490 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:09.152786 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:42:09.152894 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:42:09.152974 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:09.153072 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 11:42:09.153131 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 11:42:09.155393 11901 raft_consensus.cc:1261] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Refusing update from remote peer 81423e6a25074bde8f9ff36fe40433bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:09.155652 11697 raft_consensus.cc:1261] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Refusing update from remote peer 81423e6a25074bde8f9ff36fe40433bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:09.156029 12031 consensus_queue.cc:1035] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:09.156198 12020 consensus_queue.cc:1035] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:09.163936 11263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:09.164072 11263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:09.165966 11263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: Sending DeleteTablet for 3 replicas of tablet 3b5860c55ebb432f8ab1f8583e05fd63
I20260806 11:42:09.166425 11462 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5860c55ebb432f8ab1f8583e05fd63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:09 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:49208
I20260806 11:42:09.166512 11671 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5860c55ebb432f8ab1f8583e05fd63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:09 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:41448
I20260806 11:42:09.166596 12042 tablet_replica.cc:330] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: stopping tablet replica
I20260806 11:42:09.166718 12042 raft_consensus.cc:2229] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:42:09.166786 12041 tablet_replica.cc:330] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: stopping tablet replica
I20260806 11:42:09.166882 12042 raft_consensus.cc:2258] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:09.166929 12041 raft_consensus.cc:2229] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:09.167034 12041 raft_consensus.cc:2258] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:09.167184 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5860c55ebb432f8ab1f8583e05fd63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:09 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:47308
I20260806 11:42:09.167346 12040 tablet_replica.cc:330] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: stopping tablet replica
I20260806 11:42:09.167471 12040 raft_consensus.cc:2229] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:09.167596 12040 raft_consensus.cc:2258] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:09.167675 11263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:42:09.168095 11263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:42:09.168396 12040 ts_tablet_manager.cc:1905] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:09.168396 12042 ts_tablet_manager.cc:1905] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:09.170871 12040 ts_tablet_manager.cc:1918] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:09.170871 12042 ts_tablet_manager.cc:1918] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:09.170943 12042 log.cc:1196] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/3b5860c55ebb432f8ab1f8583e05fd63
I20260806 11:42:09.170943 12040 log.cc:1196] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/3b5860c55ebb432f8ab1f8583e05fd63
I20260806 11:42:09.171211 12040 ts_tablet_manager.cc:1939] T 3b5860c55ebb432f8ab1f8583e05fd63 P 65a1763226cf462aa521fdb39d5f9f0d: Deleting consensus metadata
I20260806 11:42:09.171211 12042 ts_tablet_manager.cc:1939] T 3b5860c55ebb432f8ab1f8583e05fd63 P 81423e6a25074bde8f9ff36fe40433bd: Deleting consensus metadata
I20260806 11:42:09.171839 11244 catalog_manager.cc:4841] TS 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757): tablet 3b5860c55ebb432f8ab1f8583e05fd63 (table test-table [id=adf9e28e3e164beba8980622fb15dc42]) successfully deleted
I20260806 11:42:09.172005 11243 catalog_manager.cc:4841] TS 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1:46609): tablet 3b5860c55ebb432f8ab1f8583e05fd63 (table test-table [id=adf9e28e3e164beba8980622fb15dc42]) successfully deleted
I20260806 11:42:09.172267 12041 ts_tablet_manager.cc:1905] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:09.174589 12041 ts_tablet_manager.cc:1918] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:09.174685 12041 log.cc:1196] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/3b5860c55ebb432f8ab1f8583e05fd63
I20260806 11:42:09.174925 12041 ts_tablet_manager.cc:1939] T 3b5860c55ebb432f8ab1f8583e05fd63 P f8ee3845af8f4483a006c82d4fa5b11d: Deleting consensus metadata
I20260806 11:42:09.175642 11243 catalog_manager.cc:4841] TS f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617): tablet 3b5860c55ebb432f8ab1f8583e05fd63 (table test-table [id=adf9e28e3e164beba8980622fb15dc42]) successfully deleted
I20260806 11:42:09.175827 11462 tablet_service.cc:1467] Processing CreateTablet for tablet 237751280ca5413fb8111825a2a19bd5 (DEFAULT_TABLE table=test-table [id=f388462bfc564a469549c17ca8c531fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:09.176083 11462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237751280ca5413fb8111825a2a19bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:09.176087 11671 tablet_service.cc:1467] Processing CreateTablet for tablet 237751280ca5413fb8111825a2a19bd5 (DEFAULT_TABLE table=test-table [id=f388462bfc564a469549c17ca8c531fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:09.176077 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 237751280ca5413fb8111825a2a19bd5 (DEFAULT_TABLE table=test-table [id=f388462bfc564a469549c17ca8c531fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:09.176309 11671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237751280ca5413fb8111825a2a19bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:09.176383 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237751280ca5413fb8111825a2a19bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:09.178213 12005 tablet_bootstrap.cc:492] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: Bootstrap starting.
I20260806 11:42:09.178401 12002 tablet_bootstrap.cc:492] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: Bootstrap starting.
I20260806 11:42:09.178445 12003 tablet_bootstrap.cc:492] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd: Bootstrap starting.
I20260806 11:42:09.179082 12005 tablet_bootstrap.cc:654] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:09.179237 12003 tablet_bootstrap.cc:654] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:09.179345 12002 tablet_bootstrap.cc:654] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:09.180387 12005 tablet_bootstrap.cc:492] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: No bootstrap required, opened a new log
I20260806 11:42:09.180487 12005 ts_tablet_manager.cc:1397] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:09.180821 12003 tablet_bootstrap.cc:492] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd: No bootstrap required, opened a new log
I20260806 11:42:09.180882 12003 ts_tablet_manager.cc:1397] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:09.180841 12005 raft_consensus.cc:348] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.180927 12005 raft_consensus.cc:374] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:09.180948 12005 raft_consensus.cc:729] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a1763226cf462aa521fdb39d5f9f0d, State: Initialized, Role: FOLLOWER
I20260806 11:42:09.181046 12005 consensus_queue.cc:260] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.181273 12003 raft_consensus.cc:348] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.181331 12005 ts_tablet_manager.cc:1428] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 11:42:09.181376 12003 raft_consensus.cc:374] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:09.181399 12003 raft_consensus.cc:729] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81423e6a25074bde8f9ff36fe40433bd, State: Initialized, Role: FOLLOWER
I20260806 11:42:09.181528 12003 consensus_queue.cc:260] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.181815 12003 ts_tablet_manager.cc:1428] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 11:42:09.182668 12002 tablet_bootstrap.cc:492] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: No bootstrap required, opened a new log
I20260806 11:42:09.182765 12002 ts_tablet_manager.cc:1397] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 11:42:09.183257 12002 raft_consensus.cc:348] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.183377 12002 raft_consensus.cc:374] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:09.183409 12002 raft_consensus.cc:729] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ee3845af8f4483a006c82d4fa5b11d, State: Initialized, Role: FOLLOWER
I20260806 11:42:09.183507 12002 consensus_queue.cc:260] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.183771 12002 ts_tablet_manager.cc:1428] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 11:42:09.194592 12031 raft_consensus.cc:482] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:42:09.194736 12031 raft_consensus.cc:504] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.195194 12031 leader_election.cc:288] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617), 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757)
I20260806 11:42:09.195611 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237751280ca5413fb8111825a2a19bd5" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a1763226cf462aa521fdb39d5f9f0d" is_pre_election: true
I20260806 11:42:09.195611 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237751280ca5413fb8111825a2a19bd5" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" is_pre_election: true
I20260806 11:42:09.195765 11901 raft_consensus.cc:2454] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 0.
I20260806 11:42:09.195765 11697 raft_consensus.cc:2454] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 0.
I20260806 11:42:09.196051 11401 leader_election.cc:302] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a1763226cf462aa521fdb39d5f9f0d, 81423e6a25074bde8f9ff36fe40433bd; no voters:
I20260806 11:42:09.196231 12031 raft_consensus.cc:2781] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:42:09.196352 12031 raft_consensus.cc:482] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:42:09.196410 12031 raft_consensus.cc:3037] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:09.197654 12031 raft_consensus.cc:504] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.198069 12031 leader_election.cc:288] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 election: Requested vote from peers f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617), 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757)
I20260806 11:42:09.198444 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237751280ca5413fb8111825a2a19bd5" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a1763226cf462aa521fdb39d5f9f0d"
I20260806 11:42:09.198477 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237751280ca5413fb8111825a2a19bd5" candidate_uuid: "81423e6a25074bde8f9ff36fe40433bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee3845af8f4483a006c82d4fa5b11d"
I20260806 11:42:09.198586 11697 raft_consensus.cc:3037] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:09.198586 11901 raft_consensus.cc:3037] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:09.199694 11697 raft_consensus.cc:2454] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 1.
I20260806 11:42:09.199694 11901 raft_consensus.cc:2454] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81423e6a25074bde8f9ff36fe40433bd in term 1.
I20260806 11:42:09.199995 11400 leader_election.cc:302] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81423e6a25074bde8f9ff36fe40433bd, f8ee3845af8f4483a006c82d4fa5b11d; no voters:
I20260806 11:42:09.200196 12031 raft_consensus.cc:2781] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:09.200304 12031 raft_consensus.cc:686] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 1 LEADER]: Becoming Leader. State: Replica: 81423e6a25074bde8f9ff36fe40433bd, State: Running, Role: LEADER
I20260806 11:42:09.200477 12031 consensus_queue.cc:237] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } }
I20260806 11:42:09.202107 11261 catalog_manager.cc:5495] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1). New cstate: current_term: 1 leader_uuid: "81423e6a25074bde8f9ff36fe40433bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81423e6a25074bde8f9ff36fe40433bd" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 46609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 } health_report { overall_health: UNKNOWN } } }
I20260806 11:42:09.226020 11697 raft_consensus.cc:1261] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Refusing update from remote peer 81423e6a25074bde8f9ff36fe40433bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:09.226020 11901 raft_consensus.cc:1261] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Refusing update from remote peer 81423e6a25074bde8f9ff36fe40433bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:09.226457 12031 consensus_queue.cc:1035] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a1763226cf462aa521fdb39d5f9f0d" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:09.226652 12013 consensus_queue.cc:1035] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ee3845af8f4483a006c82d4fa5b11d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:09.256246 11260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:09.256448 11260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:09.258285 11260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ace7ce515f742ab9b402dc03f9a8c8d: Sending DeleteTablet for 3 replicas of tablet 237751280ca5413fb8111825a2a19bd5
I20260806 11:42:09.258929 11671 tablet_service.cc:1514] Processing DeleteTablet for tablet 237751280ca5413fb8111825a2a19bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:09 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:41448
I20260806 11:42:09.259066 11462 tablet_service.cc:1514] Processing DeleteTablet for tablet 237751280ca5413fb8111825a2a19bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:09 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:49208
I20260806 11:42:09.259126 12041 tablet_replica.cc:330] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: stopping tablet replica
I20260806 11:42:09.259213 12042 tablet_replica.cc:330] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd: stopping tablet replica
I20260806 11:42:09.259172 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet 237751280ca5413fb8111825a2a19bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:09 UTC) from {username='oryx', principal='oryx/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:47308
I20260806 11:42:09.259241 12041 raft_consensus.cc:2229] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:09.259310 12042 raft_consensus.cc:2229] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:42:09.259351 12041 raft_consensus.cc:2258] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:09.259477 12042 raft_consensus.cc:2258] T 237751280ca5413fb8111825a2a19bd5 P 81423e6a25074bde8f9ff36fe40433bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:09.259639 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 11376
I20260806 11:42:09.259727 12040 tablet_replica.cc:330] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: stopping tablet replica
I20260806 11:42:09.259859 12040 raft_consensus.cc:2229] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:09.259961 12040 raft_consensus.cc:2258] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:09.265050 12040 ts_tablet_manager.cc:1905] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:09.268428 12040 ts_tablet_manager.cc:1918] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:42:09.268507 12040 log.cc:1196] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/237751280ca5413fb8111825a2a19bd5
I20260806 11:42:09.268735 12040 ts_tablet_manager.cc:1939] T 237751280ca5413fb8111825a2a19bd5 P 65a1763226cf462aa521fdb39d5f9f0d: Deleting consensus metadata
I20260806 11:42:09.269590 12041 ts_tablet_manager.cc:1905] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:09.269577 11244 catalog_manager.cc:4841] TS 65a1763226cf462aa521fdb39d5f9f0d (127.0.113.3:41757): tablet 237751280ca5413fb8111825a2a19bd5 (table test-table [id=f388462bfc564a469549c17ca8c531fb]) successfully deleted
I20260806 11:42:09.271869 12041 ts_tablet_manager.cc:1918] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 11:42:09.271997 12041 log.cc:1196] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/237751280ca5413fb8111825a2a19bd5
I20260806 11:42:09.272271 12041 ts_tablet_manager.cc:1939] T 237751280ca5413fb8111825a2a19bd5 P f8ee3845af8f4483a006c82d4fa5b11d: Deleting consensus metadata
I20260806 11:42:09.272922 11243 catalog_manager.cc:4841] TS f8ee3845af8f4483a006c82d4fa5b11d (127.0.113.2:33617): tablet 237751280ca5413fb8111825a2a19bd5 (table test-table [id=f388462bfc564a469549c17ca8c531fb]) successfully deleted
W20260806 11:42:09.276810 11243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:46609 (error 108)
I20260806 11:42:09.277086 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 11575
I20260806 11:42:09.284737 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:09.280908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:46609 (local address 127.0.0.1:49230)
0806 11:42:09.284548 (+ 3640us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:09.284636 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3557,"thread_start_us":85,"threads_started":1}
W20260806 11:42:09.284911 11243 catalog_manager.cc:4584] TS 81423e6a25074bde8f9ff36fe40433bd (127.0.113.1:46609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 237751280ca5413fb8111825a2a19bd5: Network error: Client connection negotiation failed: client connection to 127.0.113.1:46609: connect: Connection refused (error 111)
I20260806 11:42:09.286429 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 11784
I20260806 11:42:09.299856 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 11219
2026-08-06T11:42:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3205 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:42:09 dist-test-slave-45dx krb5kdc[12139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:42:09 dist-test-slave-45dx krb5kdc[12139](info): set up 2 sockets
Aug 06 11:42:09 dist-test-slave-45dx krb5kdc[12139](info): commencing operation
krb5kdc: starting...
W20260806 11:42:10.514724 452 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:42:10 dist-test-slave-45dx krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:42:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:42:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:42:10.647812 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:42587
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:42587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:10.760882 12157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:10.761206 12157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:10.761303 12157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:10.765415 12157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:10.765501 12157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:10.765538 12157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:10.765599 12157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:10.765653 12157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:10.771067 12157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:42587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:42587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:10.772214 12157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:10.773305 12157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:10.782517 12169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:10.782500 12166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:10.782527 12167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:10.782756 12157 server_base.cc:1034] running on GCE node
I20260806 11:42:10.783453 12157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:10.784706 12157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:10.785892 12157 hybrid_clock.cc:648] HybridClock initialized: now 1786016530785853 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T11:42:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1489 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:42:10 dist-test-slave-45dx krb5kdc[12176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:42:10 dist-test-slave-45dx krb5kdc[12176](info): set up 2 sockets
Aug 06 11:42:10 dist-test-slave-45dx krb5kdc[12176](info): commencing operation
krb5kdc: starting...
W20260806 11:42:12.041491 452 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:42:12 dist-test-slave-45dx krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:42:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:42:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:42:12.176041 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:35857
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:35857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:12.293918 12194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:12.294229 12194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:12.294312 12194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:12.298276 12194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:12.298372 12194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:12.298409 12194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:12.298488 12194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:12.298524 12194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:12.304011 12194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:35857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:35857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:12.305135 12194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:12.306161 12194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:12.315066 12202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:12.315140 12194 server_base.cc:1034] running on GCE node
W20260806 11:42:12.315078 12207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:12.315078 12200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:12.315918 12194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:12.317024 12194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:12.318243 12194 hybrid_clock.cc:648] HybridClock initialized: now 1786016532318219 us; error 30 us; skew 500 ppm
Aug 06 11:42:12 dist-test-slave-45dx krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:12.325942 12194 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:42:12.326934 12194 webserver.cc:460] Webserver started at http://127.0.113.62:45497/ using document root <none> and password file <none>
I20260806 11:42:12.327523 12194 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:12.327618 12194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:12.327872 12194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:12.330044 12194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "a93498f36a8543f583a4c7bf43195e54"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "c2ca26c2ab5691a3597e4fc86f9d3d39"
server_key_iv: "cc871d58babba4736e6f21c3d16ce397"
server_key_version: "encryptionkey@0"
I20260806 11:42:12.330627 12194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "a93498f36a8543f583a4c7bf43195e54"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "c2ca26c2ab5691a3597e4fc86f9d3d39"
server_key_iv: "cc871d58babba4736e6f21c3d16ce397"
server_key_version: "encryptionkey@0"
I20260806 11:42:12.334434 12194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 11:42:12.336925 12215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:12.338145 12194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:12.338292 12194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "a93498f36a8543f583a4c7bf43195e54"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "c2ca26c2ab5691a3597e4fc86f9d3d39"
server_key_iv: "cc871d58babba4736e6f21c3d16ce397"
server_key_version: "encryptionkey@0"
I20260806 11:42:12.338410 12194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:12.360033 12194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:12.363035 12194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:12.363245 12194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:12.370535 12194 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:35857
I20260806 11:42:12.370561 12300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:35857 every 8 connection(s)
I20260806 11:42:12.371492 12194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:42:12.374028 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 12194
I20260806 11:42:12.374190 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:42:12.374469 12301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:12.374531 452 external_mini_cluster.cc:1428] Setting key e8e00ce8817cbb89735465e245b71713
I20260806 11:42:12.380780 12301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: Bootstrap starting.
I20260806 11:42:12.383100 12301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: Neither blocks nor log segments found. Creating new log.
Aug 06 11:42:12 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:42:12.383875 12301 log.cc:824] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:12.385993 12301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: No bootstrap required, opened a new log
I20260806 11:42:12.388558 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:12.375924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.0.113.62:35857)
0806 11:42:12.376394 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:12.376409 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:12.376443 (+ 34us) server_negotiation.cc:407] Connection header received
0806 11:42:12.377192 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:12.377210 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:12.377528 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:12.377845 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:12.378690 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.379662 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.380618 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.381142 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.381308 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:12.383912 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:12.383917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:12.383930 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:12.383984 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:12.385721 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:12.386244 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:12.386250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:12.386255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:12.386368 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:12.386653 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:12.386658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:12.386659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:12.387047 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:12.387258 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:12.387523 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 11:42:12.387800 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260806 11:42:12.389897 12301 raft_consensus.cc:348] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } }
I20260806 11:42:12.390129 12301 raft_consensus.cc:374] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:12.390201 12301 raft_consensus.cc:729] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93498f36a8543f583a4c7bf43195e54, State: Initialized, Role: FOLLOWER
I20260806 11:42:12.390646 12301 consensus_queue.cc:260] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } }
I20260806 11:42:12.390771 12301 raft_consensus.cc:388] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:42:12.390857 12301 raft_consensus.cc:482] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:42:12.390990 12301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:12.391901 12301 raft_consensus.cc:504] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } }
I20260806 11:42:12.392251 12301 leader_election.cc:302] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a93498f36a8543f583a4c7bf43195e54; no voters:
I20260806 11:42:12.392632 12301 leader_election.cc:288] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:42:12.392776 12308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:12.393024 12308 raft_consensus.cc:686] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 1 LEADER]: Becoming Leader. State: Replica: a93498f36a8543f583a4c7bf43195e54, State: Running, Role: LEADER
I20260806 11:42:12.393409 12308 consensus_queue.cc:237] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } }
I20260806 11:42:12.393951 12301 sys_catalog.cc:564] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:42:12.395099 12312 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a93498f36a8543f583a4c7bf43195e54. Latest consensus state: current_term: 1 leader_uuid: "a93498f36a8543f583a4c7bf43195e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } } }
I20260806 11:42:12.395226 12312 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:12.395367 12311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a93498f36a8543f583a4c7bf43195e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } } }
I20260806 11:42:12.395552 12311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:12.395856 12320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:42:12.399183 12320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:42:12.405208 12320 catalog_manager.cc:1347] Generated new cluster ID: 2c021479524e4677bad10be97fa3f137
I20260806 11:42:12.405294 12320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:42:12.425930 12320 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:42:12.427245 12320 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:42:12.434340 12320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: Generated new TSK 0
I20260806 11:42:12.435139 12320 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:42:12.469240 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:35857
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:12.584127 12335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:12.584457 12335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:12.584534 12335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:12.588578 12335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:12.588657 12335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:12.588747 12335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:42:12.593892 12335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:12.595067 12335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:12.596195 12335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:12.605990 12350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:12.606069 12344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:12.606087 12343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:12.606237 12335 server_base.cc:1034] running on GCE node
I20260806 11:42:12.606769 12335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:12.607416 12335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:12.608578 12335 hybrid_clock.cc:648] HybridClock initialized: now 1786016532608555 us; error 33 us; skew 500 ppm
Aug 06 11:42:12 dist-test-slave-45dx krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:12.611431 12335 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:42:12.612274 12335 webserver.cc:460] Webserver started at http://127.0.113.1:36973/ using document root <none> and password file <none>
I20260806 11:42:12.612890 12335 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:12.612948 12335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:12.613111 12335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:12.615103 12335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "b0f17d64978941e585e26a05d91c84e6"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "349da7690a1a1f2df66e9c69cdddb400"
server_key_iv: "f9b956944dbf2bc9a2125eeb8d720a27"
server_key_version: "encryptionkey@0"
I20260806 11:42:12.615625 12335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "b0f17d64978941e585e26a05d91c84e6"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "349da7690a1a1f2df66e9c69cdddb400"
server_key_iv: "f9b956944dbf2bc9a2125eeb8d720a27"
server_key_version: "encryptionkey@0"
I20260806 11:42:12.619306 12335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:42:12.621735 12364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:12.623000 12335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:12.623142 12335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "b0f17d64978941e585e26a05d91c84e6"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "349da7690a1a1f2df66e9c69cdddb400"
server_key_iv: "f9b956944dbf2bc9a2125eeb8d720a27"
server_key_version: "encryptionkey@0"
I20260806 11:42:12.623258 12335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:12.650791 12335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:12.653848 12335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:12.654069 12335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:12.654696 12335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:12.655548 12335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:12.655596 12335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:12.655668 12335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:12.655700 12335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:12.664846 12335 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:40455
I20260806 11:42:12.664871 12538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:40455 every 8 connection(s)
I20260806 11:42:12.665822 12335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:42:12.666760 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 12335
I20260806 11:42:12.666906 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:42:12.667207 452 external_mini_cluster.cc:1428] Setting key 1eb78d4320303507dc44b643e7f79e2a
Aug 06 11:42:12 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:42:12.683213 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:12.668264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:41347 (local address 127.0.113.62:35857)
0806 11:42:12.668424 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:12.668427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:12.669564 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 11:42:12.671578 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:12.671583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:12.671646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:12.671749 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:12.673408 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.674321 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.675724 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.676188 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.676353 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:12.679074 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:12.679077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:12.679080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:12.679106 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:12.680801 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:12.681448 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:12.681451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:12.681453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:12.681548 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:12.682003 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:12.682006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:12.682007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:12.682202 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:12.682285 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:12.682972 (+ 687us) server_negotiation.cc:299] Negotiation successful
0806 11:42:12.683084 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 11:42:12.684285 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:12.668841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:41347)
0806 11:42:12.669376 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:12.669441 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:12.671360 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:12.671997 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:12.672008 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:12.672602 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:12.673244 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:12.673259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.674529 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.674533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:12.675566 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:12.675577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.676450 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.676453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:12.676614 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:12.677295 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:12.677312 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:12.678888 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:12.680950 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:12.680958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:12.680975 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:12.681316 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:12.681650 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:12.681656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:12.681659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:12.681874 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:12.682313 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:12.682323 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:12.682693 (+ 370us) client_negotiation.cc:769] Sending connection context
0806 11:42:12.682912 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 11:42:12.683302 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":116,"threads_started":1}
I20260806 11:42:12.685667 12540 heartbeater.cc:347] Connected to a master server at 127.0.113.62:35857
I20260806 11:42:12.685956 12540 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:12.686523 12540 heartbeater.cc:511] Master 127.0.113.62:35857 requested a full tablet report, sending...
I20260806 11:42:12.688325 12241 ts_manager.cc:194] Registered new tserver with Master: b0f17d64978941e585e26a05d91c84e6 (127.0.113.1:40455)
I20260806 11:42:12.690016 12241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:41347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:42:12.699815 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:35857
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:12.815467 12551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:12.815771 12551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:12.815855 12551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:12.820015 12551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:12.820103 12551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:12.820194 12551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:42:12.826211 12551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:12.827360 12551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:12.828485 12551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:12.838047 12558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:12.838075 12559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:12.838078 12561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:12.838423 12551 server_base.cc:1034] running on GCE node
I20260806 11:42:12.838800 12551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:12.839421 12551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:12.840601 12551 hybrid_clock.cc:648] HybridClock initialized: now 1786016532840574 us; error 48 us; skew 500 ppm
Aug 06 11:42:12 dist-test-slave-45dx krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:12.843498 12551 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:42:12.844453 12551 webserver.cc:460] Webserver started at http://127.0.113.2:40387/ using document root <none> and password file <none>
I20260806 11:42:12.845024 12551 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:12.845120 12551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:12.845351 12551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:12.847314 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "9790aaab31a0436aaacffe2cebe9a77f"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "18a775d8c67e5eb61af049a3a116d37b"
server_key_iv: "21e52625b2ae1ffef45d58c7c94187a4"
server_key_version: "encryptionkey@0"
I20260806 11:42:12.847899 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "9790aaab31a0436aaacffe2cebe9a77f"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "18a775d8c67e5eb61af049a3a116d37b"
server_key_iv: "21e52625b2ae1ffef45d58c7c94187a4"
server_key_version: "encryptionkey@0"
I20260806 11:42:12.851485 12551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260806 11:42:12.853919 12570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:12.855065 12551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:12.855199 12551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "9790aaab31a0436aaacffe2cebe9a77f"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "18a775d8c67e5eb61af049a3a116d37b"
server_key_iv: "21e52625b2ae1ffef45d58c7c94187a4"
server_key_version: "encryptionkey@0"
I20260806 11:42:12.855310 12551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:12.876165 12551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:12.879560 12551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:12.879768 12551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:12.880399 12551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:12.881243 12551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:12.881315 12551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:12.881393 12551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:12.881439 12551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:12.890654 12551 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:35965
I20260806 11:42:12.890683 12729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:35965 every 8 connection(s)
I20260806 11:42:12.891502 12551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:42:12.898106 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 12551
I20260806 11:42:12.898239 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:42:12.898523 452 external_mini_cluster.cc:1428] Setting key 328d5ff2ec54749c30da63898b3cf951
Aug 06 11:42:12 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:42:12.906323 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:12.893234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:34975 (local address 127.0.113.62:35857)
0806 11:42:12.893399 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:12.893403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:12.894274 (+ 871us) server_negotiation.cc:407] Connection header received
0806 11:42:12.895886 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:12.895889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:12.895937 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:12.896013 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:12.897208 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.897738 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.899053 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.899439 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.899545 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:12.902433 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:12.902437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:12.902440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:12.902472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:12.903936 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:12.904592 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:12.904598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:12.904603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:12.904712 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:12.905076 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:12.905082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:12.905086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:12.905353 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:12.905483 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:12.905933 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 11:42:12.906079 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 11:42:12.906857 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:12.893617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:34975)
0806 11:42:12.894090 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:12.894140 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:12.895676 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:12.896173 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:12.896179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:12.896617 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:12.897074 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:12.897087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.897851 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.897853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:12.898883 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:12.898895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:12.899596 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:12.899602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:12.899846 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:12.900596 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:12.900612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:12.902263 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:12.904069 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:12.904075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:12.904087 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:12.904481 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:12.904819 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:12.904822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:12.904824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:12.904957 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:12.905473 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:12.905478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:12.905683 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 11:42:12.905824 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 11:42:12.906048 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260806 11:42:12.907948 12731 heartbeater.cc:347] Connected to a master server at 127.0.113.62:35857
I20260806 11:42:12.908206 12731 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:12.908815 12731 heartbeater.cc:511] Master 127.0.113.62:35857 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:42:12.909891 12241 ts_manager.cc:194] Registered new tserver with Master: 9790aaab31a0436aaacffe2cebe9a77f (127.0.113.2:35965)
I20260806 11:42:12.910490 12241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:34975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:42:12.927253 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:35857
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:13.038800 12739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:13.039090 12739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:13.039196 12739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:13.043455 12739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:13.043531 12739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:13.043630 12739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:42:13.048669 12739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:13.049683 12739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:13.050678 12739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:13.060364 12751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:13.060354 12747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:13.060536 12739 server_base.cc:1034] running on GCE node
W20260806 11:42:13.060333 12748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:13.061041 12739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:13.061699 12739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:13.062860 12739 hybrid_clock.cc:648] HybridClock initialized: now 1786016533062827 us; error 47 us; skew 500 ppm
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:13.065632 12739 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:42:13.066488 12739 webserver.cc:460] Webserver started at http://127.0.113.3:40135/ using document root <none> and password file <none>
I20260806 11:42:13.066999 12739 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:13.067058 12739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:13.067221 12739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:13.069254 12739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "de16811bfac94ac78260c0663d102cff"
format_stamp: "Formatted at 2026-08-06 11:42:13 on dist-test-slave-45dx"
server_key: "c3483dc64d7fb0a945b88be017adb2d4"
server_key_iv: "38577790790147863361d56c7956317d"
server_key_version: "encryptionkey@0"
I20260806 11:42:13.069769 12739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "de16811bfac94ac78260c0663d102cff"
format_stamp: "Formatted at 2026-08-06 11:42:13 on dist-test-slave-45dx"
server_key: "c3483dc64d7fb0a945b88be017adb2d4"
server_key_iv: "38577790790147863361d56c7956317d"
server_key_version: "encryptionkey@0"
I20260806 11:42:13.073246 12739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:42:13.075464 12762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.076651 12739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:13.076781 12739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "de16811bfac94ac78260c0663d102cff"
format_stamp: "Formatted at 2026-08-06 11:42:13 on dist-test-slave-45dx"
server_key: "c3483dc64d7fb0a945b88be017adb2d4"
server_key_iv: "38577790790147863361d56c7956317d"
server_key_version: "encryptionkey@0"
I20260806 11:42:13.076891 12739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:13.094280 12739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:13.097260 12739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:13.097473 12739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:13.098070 12739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:13.098901 12739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:13.098974 12739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.099048 12739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:13.099092 12739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.108354 12739 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:46565
I20260806 11:42:13.108428 12916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:46565 every 8 connection(s)
I20260806 11:42:13.109339 12739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:42:13.113598 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 12739
I20260806 11:42:13.113745 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:42:13.114171 452 external_mini_cluster.cc:1428] Setting key e96217ec67559a836f92a1ca3d8798fe
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:42:13.123905 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:13.111319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:54609 (local address 127.0.113.62:35857)
0806 11:42:13.111507 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:13.111511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:13.112343 (+ 832us) server_negotiation.cc:407] Connection header received
0806 11:42:13.114039 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:13.114042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:13.114091 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:13.114166 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:13.115569 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.116182 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.117431 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.117651 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.117764 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.120339 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:13.120349 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:13.120351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:13.120394 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:13.121840 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:13.122382 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:13.122388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:13.122393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:13.122481 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:13.122827 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:13.122832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:13.122834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:13.123027 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:13.123134 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:13.123615 (+ 481us) server_negotiation.cc:299] Negotiation successful
0806 11:42:13.123732 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 11:42:13.124696 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:13.111664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:54609)
0806 11:42:13.112173 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:13.112218 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:13.113801 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:13.114394 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:13.114403 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:13.114804 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:13.115388 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.115413 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.116354 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.116358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.117284 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.117294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.117804 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.117808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.117976 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.118607 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:13.118623 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:13.120147 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:13.121982 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:13.121988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:13.122002 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:13.122270 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:13.122579 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:13.122584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:13.122586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:13.122727 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:13.123134 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:13.123141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:13.123379 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 11:42:13.123559 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 11:42:13.123860 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260806 11:42:13.125815 12917 heartbeater.cc:347] Connected to a master server at 127.0.113.62:35857
I20260806 11:42:13.126065 12917 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:13.126564 12917 heartbeater.cc:511] Master 127.0.113.62:35857 requested a full tablet report, sending...
I20260806 11:42:13.127718 12241 ts_manager.cc:194] Registered new tserver with Master: de16811bfac94ac78260c0663d102cff (127.0.113.3:46565)
I20260806 11:42:13.128373 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:42:13.128563 12241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:54609
WARNING: no policy specified for oryx/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab.
Entry for principal oryx/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab.
WARNING: no policy specified for oryx/127.0.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab.
Entry for principal oryx/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab.
WARNING: no policy specified for oryx/127.0.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab.
Entry for principal oryx/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab.
I20260806 11:42:13.203660 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 12335
I20260806 11:42:13.226401 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 12551
I20260806 11:42:13.238793 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 12739
I20260806 11:42:13.248780 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 12194
I20260806 11:42:13.258019 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:35857
--webserver_interface=127.0.113.62
--webserver_port=45497
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:35857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:13.371691 12932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:13.371953 12932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:13.371991 12932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:13.376042 12932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:13.376101 12932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:13.376121 12932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:13.376137 12932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:13.376152 12932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:13.381371 12932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:35857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:35857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=45497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:13.382484 12932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:13.383538 12932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:13.391965 12944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:13.391960 12940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:13.392019 12941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:13.392342 12932 server_base.cc:1034] running on GCE node
I20260806 11:42:13.392946 12932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:13.394007 12932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:13.395202 12932 hybrid_clock.cc:648] HybridClock initialized: now 1786016533395176 us; error 38 us; skew 500 ppm
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:13.398030 12932 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:42:13.398931 12932 webserver.cc:460] Webserver started at http://127.0.113.62:45497/ using document root <none> and password file <none>
I20260806 11:42:13.399446 12932 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:13.399519 12932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:13.403282 12932 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:13.405400 12952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.406709 12932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 11:42:13.406869 12932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "a93498f36a8543f583a4c7bf43195e54"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "c2ca26c2ab5691a3597e4fc86f9d3d39"
server_key_iv: "cc871d58babba4736e6f21c3d16ce397"
server_key_version: "encryptionkey@0"
I20260806 11:42:13.407387 12932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:13.422605 12932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:13.425639 12932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:13.425853 12932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:13.433197 12932 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:35857
I20260806 11:42:13.433223 13024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:35857 every 8 connection(s)
I20260806 11:42:13.434249 12932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:42:13.434717 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 12932
I20260806 11:42:13.435279 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:40455
--local_ip_for_outbound_sockets=127.0.113.1
--tserver_master_addrs=127.0.113.62:35857
--webserver_port=36973
--webserver_interface=127.0.113.1
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.113.1
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:42:13.438817 13025 sys_catalog.cc:263] Verifying existing consensus state
I20260806 11:42:13.441490 13025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: Bootstrap starting.
I20260806 11:42:13.464764 13025 log.cc:824] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:13.471079 13025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:42:13.471467 13025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: Bootstrap complete.
I20260806 11:42:13.475663 13025 raft_consensus.cc:348] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } }
I20260806 11:42:13.476044 13025 raft_consensus.cc:729] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93498f36a8543f583a4c7bf43195e54, State: Initialized, Role: FOLLOWER
I20260806 11:42:13.476631 13025 consensus_queue.cc:260] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } }
I20260806 11:42:13.476749 13025 raft_consensus.cc:388] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:42:13.476838 13025 raft_consensus.cc:482] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:42:13.476936 13025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 1 FOLLOWER]: Advancing to term 2
I20260806 11:42:13.478855 13025 raft_consensus.cc:504] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } }
I20260806 11:42:13.479194 13025 leader_election.cc:302] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a93498f36a8543f583a4c7bf43195e54; no voters:
I20260806 11:42:13.479544 13025 leader_election.cc:288] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 11:42:13.479626 13029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 11:42:13.479907 13029 raft_consensus.cc:686] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [term 2 LEADER]: Becoming Leader. State: Replica: a93498f36a8543f583a4c7bf43195e54, State: Running, Role: LEADER
I20260806 11:42:13.480252 13029 consensus_queue.cc:237] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } }
I20260806 11:42:13.480842 13025 sys_catalog.cc:564] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:42:13.481788 13030 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a93498f36a8543f583a4c7bf43195e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } } }
I20260806 11:42:13.481840 13032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a93498f36a8543f583a4c7bf43195e54. Latest consensus state: current_term: 2 leader_uuid: "a93498f36a8543f583a4c7bf43195e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93498f36a8543f583a4c7bf43195e54" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 35857 } } }
I20260806 11:42:13.481921 13030 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:13.481957 13032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:13.483325 13045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:42:13.486362 13045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:42:13.486987 13045 catalog_manager.cc:1259] Loaded cluster ID: 2c021479524e4677bad10be97fa3f137
I20260806 11:42:13.487053 13045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:42:13.489425 13045 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:42:13.490089 13045 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a93498f36a8543f583a4c7bf43195e54: Loaded TSK: 0
I20260806 11:42:13.491254 13045 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 11:42:13.558346 13027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:13.558660 13027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:13.558748 13027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:13.562845 13027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:13.562924 13027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:13.563010 13027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:42:13.568393 13027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.1.keytab
--principal=oryx/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:40455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=36973
--tserver_master_addrs=127.0.113.62:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:13.569401 13027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:13.570390 13027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:13.579603 13064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:13.579773 13066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:13.579628 13061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:13.580327 13027 server_base.cc:1034] running on GCE node
I20260806 11:42:13.580785 13027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:13.581596 13027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:13.582803 13027 hybrid_clock.cc:648] HybridClock initialized: now 1786016533582766 us; error 44 us; skew 500 ppm
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016533, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:13.586256 13027 init.cc:377] Logged in from keytab as oryx/127.0.113.1@KRBTEST.COM (short username oryx)
I20260806 11:42:13.587342 13027 webserver.cc:460] Webserver started at http://127.0.113.1:36973/ using document root <none> and password file <none>
I20260806 11:42:13.588012 13027 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:13.588127 13027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:13.593291 13027 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:42:13.595767 13073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.597294 13027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:42:13.597415 13027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "b0f17d64978941e585e26a05d91c84e6"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "349da7690a1a1f2df66e9c69cdddb400"
server_key_iv: "f9b956944dbf2bc9a2125eeb8d720a27"
server_key_version: "encryptionkey@0"
I20260806 11:42:13.598114 13027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:13.618069 13027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:13.621582 13027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:13.621804 13027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:13.622462 13027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:13.623344 13027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:13.623392 13027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.623463 13027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:13.623494 13027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.634574 13027 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:40455
I20260806 11:42:13.634613 13236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:40455 every 8 connection(s)
I20260806 11:42:13.635543 13027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:42:13.643452 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 13027
I20260806 11:42:13.643970 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:35965
--local_ip_for_outbound_sockets=127.0.113.2
--tserver_master_addrs=127.0.113.62:35857
--webserver_port=40387
--webserver_interface=127.0.113.2
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.113.2
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 11:42:13.649011 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:13.637635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:55069 (local address 127.0.113.62:35857)
0806 11:42:13.638093 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:13.638114 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:13.638425 (+ 311us) server_negotiation.cc:407] Connection header received
0806 11:42:13.639901 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:13.639912 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:13.640180 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:13.640502 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:13.641527 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.642469 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.643770 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.644371 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.644501 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.648289 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:55069: BlockingRecv error: recv got EOF from 127.0.113.1:55069 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":96,"threads_started":1}
I20260806 11:42:13.649019 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:13.637786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:55069)
0806 11:42:13.638268 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:13.638308 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:13.639554 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:13.640596 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:13.640602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:13.640941 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:13.641373 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.641387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.642585 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.642587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.643648 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.643659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.644537 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.644544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.644778 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.645409 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:13.645435 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:13.648079 (+ 2644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":147,"threads_started":1}
W20260806 11:42:13.649475 13237 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:35857 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.113.62:35857: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:13.655620 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:13.649922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:43127)
0806 11:42:13.650032 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:13.650049 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:13.650122 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:13.650395 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:13.650397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:13.650534 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:13.650619 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.650623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.651696 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.651698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.652216 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.652222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.652906 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.652909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.652983 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.653524 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:13.653536 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:13.655477 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 11:42:13.655809 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:13.650030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:43127 (local address 127.0.113.62:35857)
0806 11:42:13.650158 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:13.650164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:13.650181 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:42:13.650224 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:13.650229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:13.650284 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:13.650386 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:13.650735 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.651588 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.652350 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.652758 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.652909 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.655660 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:43127: BlockingRecv error: recv got EOF from 127.0.113.1:43127 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:13.661381 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:13.656152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:52477)
0806 11:42:13.656351 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:13.656368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:13.656428 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:13.656709 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:13.656711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:13.656839 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:13.656902 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.656906 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.657962 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.657964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.658459 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.658466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.659042 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.659047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.659133 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.659589 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:13.659600 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:13.661296 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260806 11:42:13.661542 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:13.656199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:52477 (local address 127.0.113.62:35857)
0806 11:42:13.656373 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:13.656380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:13.656418 (+ 38us) server_negotiation.cc:407] Connection header received
0806 11:42:13.656530 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:13.656536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:13.656593 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:13.656681 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:13.657014 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.657842 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.658565 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.658912 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.659027 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.661443 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:52477: BlockingRecv error: recv got EOF from 127.0.113.1:52477 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 11:42:13.661567 13237 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 11:42:13.757478 13241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:13.757717 13241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:13.757757 13241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:13.761684 13241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:13.761735 13241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:13.761806 13241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:42:13.766770 13241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.2.keytab
--principal=oryx/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:35965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=40387
--tserver_master_addrs=127.0.113.62:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:13.767659 13241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:13.768620 13241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:13.778002 13248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:13.778007 13251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:13.778012 13249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:13.778187 13241 server_base.cc:1034] running on GCE node
I20260806 11:42:13.778620 13241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:13.779277 13241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:13.780465 13241 hybrid_clock.cc:648] HybridClock initialized: now 1786016533780446 us; error 28 us; skew 500 ppm
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016533, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:13.783282 13241 init.cc:377] Logged in from keytab as oryx/127.0.113.2@KRBTEST.COM (short username oryx)
I20260806 11:42:13.784137 13241 webserver.cc:460] Webserver started at http://127.0.113.2:40387/ using document root <none> and password file <none>
I20260806 11:42:13.784713 13241 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:13.784823 13241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:13.788388 13241 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260806 11:42:13.790409 13261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.791527 13241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 11:42:13.791658 13241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "9790aaab31a0436aaacffe2cebe9a77f"
format_stamp: "Formatted at 2026-08-06 11:42:12 on dist-test-slave-45dx"
server_key: "18a775d8c67e5eb61af049a3a116d37b"
server_key_iv: "21e52625b2ae1ffef45d58c7c94187a4"
server_key_version: "encryptionkey@0"
I20260806 11:42:13.792183 13241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:13.810544 13241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:13.814234 13241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:13.814450 13241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:13.815043 13241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:13.815886 13241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:13.815932 13241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.815996 13241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:13.816036 13241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.824988 13241 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:35965
I20260806 11:42:13.825013 13416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:35965 every 8 connection(s)
I20260806 11:42:13.825959 13241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:42:13.831612 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 13241
I20260806 11:42:13.832082 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:46565
--local_ip_for_outbound_sockets=127.0.113.3
--tserver_master_addrs=127.0.113.62:35857
--webserver_port=40135
--webserver_interface=127.0.113.3
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.113.3
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 11:42:13.837179 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:13.827714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:34577 (local address 127.0.113.62:35857)
0806 11:42:13.827849 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:13.827852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:13.828636 (+ 784us) server_negotiation.cc:407] Connection header received
0806 11:42:13.830017 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:13.830020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:13.830085 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:13.830190 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:13.831313 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.831894 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.833114 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.833334 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.833471 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.837076 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:34577: BlockingRecv error: recv got EOF from 127.0.113.2:34577 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:42:13.838024 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:13.828028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:34577)
0806 11:42:13.828489 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:13.828530 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:13.829807 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:13.830367 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:13.830374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:13.830717 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:13.831162 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.831174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.832016 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.832020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.832985 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.832995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.833523 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.833527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.833691 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.834284 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:13.834299 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:13.836955 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
W20260806 11:42:13.838490 13417 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:35857 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.113.62:35857: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 11:42:13.844791 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:13.839053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:46075 (local address 127.0.113.62:35857)
0806 11:42:13.839173 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:13.839176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:13.839188 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:42:13.839289 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:13.839291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:13.839325 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:13.839391 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:13.839919 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.840429 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.841675 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.841894 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.841966 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.844698 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:46075: BlockingRecv error: recv got EOF from 127.0.113.2:46075 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 11:42:13.844786 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:13.838924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:46075)
0806 11:42:13.839085 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:13.839103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:13.839196 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:13.839450 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:13.839455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:13.839650 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:13.839793 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.839804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.840536 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.840542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.841552 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.841567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.842008 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.842015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.842141 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.842612 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:13.842631 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:13.844602 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:13.850328 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:13.845475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:49961)
0806 11:42:13.845723 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:13.845737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:13.845838 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:13.846075 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:13.846078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:13.846229 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:13.846291 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.846296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.847006 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.847011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.847541 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:13.847549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.847983 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.847986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:13.848064 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.848572 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:13.848584 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:13.850199 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187}
W20260806 11:42:13.850410 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:13.845588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:49961 (local address 127.0.113.62:35857)
0806 11:42:13.845704 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:13.845708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:13.845802 (+ 94us) server_negotiation.cc:407] Connection header received
0806 11:42:13.845935 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:13.845939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:13.845981 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:13.846060 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:13.846424 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.846891 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.847661 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:13.847873 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:13.847982 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:13.850303 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:49961: BlockingRecv error: recv got EOF from 127.0.113.2:49961 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 11:42:13.850536 13417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 11:42:13.945930 13421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:13.946158 13421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:13.946198 13421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:13.950140 13421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:13.950191 13421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:13.950287 13421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:42:13.955153 13421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/krb5kdc/oryx_127.0.113.3.keytab
--principal=oryx/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:46565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=40135
--tserver_master_addrs=127.0.113.62:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:13.956070 13421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:13.957010 13421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:13.966275 13433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:13.966303 13432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:13.966275 13435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:13.966555 13421 server_base.cc:1034] running on GCE node
I20260806 11:42:13.967010 13421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:13.967607 13421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:13.968789 13421 hybrid_clock.cc:648] HybridClock initialized: now 1786016533968776 us; error 26 us; skew 500 ppm
Aug 06 11:42:13 dist-test-slave-45dx krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016533, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:13.971478 13421 init.cc:377] Logged in from keytab as oryx/127.0.113.3@KRBTEST.COM (short username oryx)
I20260806 11:42:13.972272 13421 webserver.cc:460] Webserver started at http://127.0.113.3:40135/ using document root <none> and password file <none>
I20260806 11:42:13.972831 13421 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:13.972940 13421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:13.976580 13421 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 11:42:13.978448 13448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.979629 13421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:13.979765 13421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "de16811bfac94ac78260c0663d102cff"
format_stamp: "Formatted at 2026-08-06 11:42:13 on dist-test-slave-45dx"
server_key: "c3483dc64d7fb0a945b88be017adb2d4"
server_key_iv: "38577790790147863361d56c7956317d"
server_key_version: "encryptionkey@0"
I20260806 11:42:13.980275 13421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:13.991868 13421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:13.994928 13421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:13.995168 13421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:13.995684 13421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:13.996603 13421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:13.996649 13421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:13.996683 13421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:13.996707 13421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:14.005412 13421 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:46565
I20260806 11:42:14.005496 13603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:46565 every 8 connection(s)
I20260806 11:42:14.006322 13421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:42:14.009368 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 13421
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 11:42:14.018549 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:14.008054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:54623 (local address 127.0.113.62:35857)
0806 11:42:14.008182 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:14.008186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:14.009252 (+ 1066us) server_negotiation.cc:407] Connection header received
0806 11:42:14.011167 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:14.011169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:14.011230 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:14.011301 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:14.012590 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.013152 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.014541 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.014827 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.014941 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.018391 (+ 3450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:54623: BlockingRecv error: recv got EOF from 127.0.113.3:54623 (error 108)
Metrics: {"server-negotiator.queue_time_us":52,"spinlock_wait_cycles":2304}
I20260806 11:42:14.019250 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:14.008514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:54623)
0806 11:42:14.009039 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:14.009127 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:14.010957 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:14.011501 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:14.011509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:14.011838 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:14.012432 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.012460 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.013276 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.013278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.014407 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.014418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.015003 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.015006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.015146 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.015707 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:14.015723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:14.018293 (+ 2570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":159,"threads_started":1}
W20260806 11:42:14.019699 13604 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:35857 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.113.62:35857: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
I20260806 11:42:14.020542 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:14.010667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.0.113.62:35857)
0806 11:42:14.011089 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:14.011096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:14.011110 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:42:14.011157 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:14.011160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:14.011266 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:14.011362 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:14.012256 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.012918 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.013893 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.014250 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.014356 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.015756 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:14.015769 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:14.015783 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:14.015809 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:14.017812 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:14.018280 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:14.018285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:14.018289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:14.018405 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:14.018770 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:14.018774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:14.018775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:14.019190 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:14.019403 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:14.020154 (+ 751us) server_negotiation.cc:299] Negotiation successful
0806 11:42:14.020424 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"spinlock_wait_cycles":5376,"thread_start_us":110,"threads_started":1}
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:14.024927 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:14.020146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:42177)
0806 11:42:14.020237 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:14.020252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:14.020348 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:14.020585 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:14.020588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:14.020734 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:14.020820 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.020825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.021518 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.021523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.022064 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.022073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.022705 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.022709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.022801 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.023157 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:14.023167 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:14.024798 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260806 11:42:14.025027 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:14.020224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:42177 (local address 127.0.113.62:35857)
0806 11:42:14.020346 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:14.020350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:14.020364 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:42:14.020435 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:14.020438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:14.020487 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:14.020569 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:14.020930 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.021402 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.022378 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.022585 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.022692 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.024919 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:42177: BlockingRecv error: recv got EOF from 127.0.113.3:42177 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:14.029536 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:14.025502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:46263)
0806 11:42:14.025583 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:14.025597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:14.025680 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:14.025855 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:14.025857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:14.025982 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:14.026042 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.026046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.026657 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.026660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.027167 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.027174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.027550 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.027552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.027605 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.027951 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:14.027962 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:14.029458 (+ 1496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 11:42:14.029713 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:14.025546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:46263 (local address 127.0.113.62:35857)
0806 11:42:14.025693 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:14.025696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:14.025708 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:42:14.025735 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:14.025737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:14.025766 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:14.025828 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:14.026133 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.026573 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.027258 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.027461 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.027533 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.029592 (+ 2059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:46263: BlockingRecv error: recv got EOF from 127.0.113.3:46263 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 11:42:14.029736 13604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:14.667554 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:14.662259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:53881)
0806 11:42:14.662553 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:14.662568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:14.662666 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:14.662883 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:14.662885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:14.663077 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:14.663211 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.663217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.663953 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.663955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.664554 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.664562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.664976 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.664978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.665050 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.665527 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:14.665538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:14.667431 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
W20260806 11:42:14.667964 13614 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:14.662352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:53881 (local address 127.0.113.62:35857)
0806 11:42:14.662590 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:14.662593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:14.662620 (+ 27us) server_negotiation.cc:407] Connection header received
0806 11:42:14.662732 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:14.662734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:14.662775 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:14.662849 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:14.663326 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.663858 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.664654 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.664879 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.664959 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.667635 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:53881: BlockingRecv error: recv got EOF from 127.0.113.1:53881 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:14 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:14.856712 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:14.851189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:48291)
0806 11:42:14.851470 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:14.851486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:14.851600 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:14.851914 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:14.851918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:14.852169 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:14.852298 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.852309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.853098 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.853100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.853689 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:14.853697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.854101 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.854104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:14.854163 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.854724 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:14.854738 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:14.856595 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
W20260806 11:42:14.856839 13614 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:14.851275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:48291 (local address 127.0.113.62:35857)
0806 11:42:14.851419 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:14.851424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:14.851576 (+ 152us) server_negotiation.cc:407] Connection header received
0806 11:42:14.851701 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:14.851704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:14.851760 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:14.851841 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:14.852454 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.853008 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.853789 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:14.854007 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:14.854081 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:14.856709 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:48291: BlockingRecv error: recv got EOF from 127.0.113.2:48291 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 11:42:15 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:15 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:15.036221 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:15.030448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:57545)
0806 11:42:15.030770 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:15.030785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:15.030906 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:15.031168 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:15.031170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:15.031360 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:15.031498 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:15.031504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.032229 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.032231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:15.032889 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:15.032900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.033303 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.033306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:15.033387 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:15.033902 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:15.033914 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:15.036092 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":94,"threads_started":1}
W20260806 11:42:15.036384 13614 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:15.030539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:57545 (local address 127.0.113.62:35857)
0806 11:42:15.030700 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:15.030704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:15.030868 (+ 164us) server_negotiation.cc:407] Connection header received
0806 11:42:15.030998 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:15.031001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:15.031051 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:15.031128 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:15.031613 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.032132 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.033001 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.033206 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.033281 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:15.036231 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:57545: BlockingRecv error: recv got EOF from 127.0.113.3:57545 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 11:42:15 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:15 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:15.674331 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:15.668467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:56129)
0806 11:42:15.668739 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:15.668754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:15.668844 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:15.669098 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:15.669100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:15.669286 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:15.669380 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:15.669385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.670105 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.670108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:15.670907 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:15.670919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.671509 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.671513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:15.671597 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:15.672181 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:15.672204 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:15.674171 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
W20260806 11:42:15.674485 13621 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:15.668517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:56129 (local address 127.0.113.62:35857)
0806 11:42:15.668775 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:15.668778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:15.668795 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:42:15.668935 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:15.668937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:15.668982 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:15.669063 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:15.669483 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.670012 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.671029 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.671380 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.671505 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:15.674322 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:56129: BlockingRecv error: recv got EOF from 127.0.113.1:56129 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
Aug 06 11:42:15 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:15 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:15.863381 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:15.857636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:37959)
0806 11:42:15.857990 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:15.858008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:15.858165 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:15.858468 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:15.858471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:15.858694 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:15.858791 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:15.858797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.859616 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.859621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:15.860194 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:15.860202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.860690 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.860693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:15.860783 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:15.861296 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:15.861308 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:15.863235 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":95,"threads_started":1}
W20260806 11:42:15.863451 13621 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:15.857787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:37959 (local address 127.0.113.62:35857)
0806 11:42:15.857967 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:15.857973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:15.858104 (+ 131us) server_negotiation.cc:407] Connection header received
0806 11:42:15.858294 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:15.858297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:15.858342 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:15.858442 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:15.858943 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.859498 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.860342 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:15.860570 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:15.860684 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:15.863353 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:37959: BlockingRecv error: recv got EOF from 127.0.113.2:37959 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 11:42:16 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:16 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:16.042943 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:16.037104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:39837)
0806 11:42:16.037454 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:16.037472 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:16.037613 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:16.037885 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:16.037888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:16.038107 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:16.038203 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:16.038209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.038981 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.038988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:16.039557 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:16.039565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.040072 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.040076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:16.040173 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:16.040813 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:16.040833 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:16.042786 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
W20260806 11:42:16.043107 13621 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:16.037289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:39837 (local address 127.0.113.62:35857)
0806 11:42:16.037441 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:16.037446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:16.037552 (+ 106us) server_negotiation.cc:407] Connection header received
0806 11:42:16.037713 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:16.037717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:16.037766 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:16.037862 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:16.038325 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.038857 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.039680 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.039924 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.040042 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:16.042940 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:39837: BlockingRecv error: recv got EOF from 127.0.113.3:39837 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 11:42:16 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:16 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:16.681007 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:16.675228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:40565)
0806 11:42:16.675574 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:16.675588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:16.675684 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:16.675932 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:16.675934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:16.676125 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:16.676222 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:16.676227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.677134 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.677137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:16.677728 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:16.677736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.678198 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.678200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:16.678258 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:16.678853 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:16.678865 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:16.680865 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":99,"threads_started":1}
W20260806 11:42:16.681198 13625 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:16.675382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:40565 (local address 127.0.113.62:35857)
0806 11:42:16.675659 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:16.675663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:16.675677 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:42:16.675767 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:16.675769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:16.675812 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:16.675901 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:16.676446 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.677013 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.677845 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.678094 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.678178 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:16.681009 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:40565: BlockingRecv error: recv got EOF from 127.0.113.1:40565 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":123,"threads_started":1}
Aug 06 11:42:16 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:16 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:16.870644 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:16.864284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:60061)
0806 11:42:16.864709 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:16.864722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:16.864835 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:16.865085 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:16.865087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:16.865277 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:16.865378 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:16.865383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.866241 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.866244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:16.866811 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:16.866820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.867332 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.867335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:16.867410 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:16.867982 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:16.867999 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:16.870434 (+ 2435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
W20260806 11:42:16.870702 13625 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:16.864464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:60061 (local address 127.0.113.62:35857)
0806 11:42:16.864668 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:16.864674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:16.864791 (+ 117us) server_negotiation.cc:407] Connection header received
0806 11:42:16.864929 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:16.864932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:16.864981 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:16.865045 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:16.865526 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.866068 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.866975 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:16.867215 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:16.867301 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:16.870584 (+ 3283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:60061: BlockingRecv error: recv got EOF from 127.0.113.2:60061 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 11:42:17 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:17 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:17.049330 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:17.043823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:55271)
0806 11:42:17.044183 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:17.044198 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:17.044357 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:17.044652 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:17.044656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:17.044880 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:17.044979 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:17.045005 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.045720 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.045722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:17.046292 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:17.046299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.046702 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.046705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:17.046768 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:17.047347 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:17.047359 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:17.049215 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":138,"threads_started":1}
W20260806 11:42:17.049510 13625 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:17.043981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:55271 (local address 127.0.113.62:35857)
0806 11:42:17.044144 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:17.044148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:17.044291 (+ 143us) server_negotiation.cc:407] Connection header received
0806 11:42:17.044470 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:17.044473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:17.044522 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:17.044624 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:17.045106 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.045621 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.046394 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.046604 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.046679 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:17.049350 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:55271: BlockingRecv error: recv got EOF from 127.0.113.3:55271 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 11:42:17 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:17 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:17.695434 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:17.681952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:35121)
0806 11:42:17.682332 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:17.682350 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:17.682511 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:17.682848 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:17.682851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:17.683102 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:17.683205 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:17.683211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.684035 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.684039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:17.684596 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:17.684605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.685024 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.685030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:17.685112 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:17.693073 (+ 7961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:17.693086 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:17.695313 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":103,"threads_started":1}
W20260806 11:42:17.695710 13629 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:17.681967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:35121 (local address 127.0.113.62:35857)
0806 11:42:17.682280 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:17.682283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:17.682456 (+ 173us) server_negotiation.cc:407] Connection header received
0806 11:42:17.682620 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:17.682625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:17.682675 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:17.682826 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:17.683323 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.683899 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.684698 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.684912 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.684987 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:17.695452 (+ 10465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:35121: BlockingRecv error: recv got EOF from 127.0.113.1:35121 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":139,"threads_started":1}
Aug 06 11:42:17 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:17 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:17.877377 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:17.871462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:42569)
0806 11:42:17.871896 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:17.871911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:17.872049 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:17.872394 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:17.872397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:17.872610 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:17.872706 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:17.872712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.873530 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.873536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:17.874086 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:17.874093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.874558 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.874562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:17.874651 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:17.875411 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:17.875424 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:17.877259 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":112,"threads_started":1}
W20260806 11:42:17.877511 13629 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:17.871557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:42569 (local address 127.0.113.62:35857)
0806 11:42:17.871853 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:17.871857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:17.871996 (+ 139us) server_negotiation.cc:407] Connection header received
0806 11:42:17.872173 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:17.872178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:17.872233 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:17.872345 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:17.872834 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.873420 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.874210 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:17.874444 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:17.874551 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:17.877390 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:42569: BlockingRecv error: recv got EOF from 127.0.113.2:42569 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Aug 06 11:42:18 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:18 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:18.055735 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:18.050164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:56183)
0806 11:42:18.050498 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:18.050511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:18.050654 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:18.050948 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:18.050951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:18.051163 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:18.051256 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:18.051262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.052004 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.052009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:18.052584 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:18.052593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.053053 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.053057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:18.053143 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:18.053669 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:18.053681 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:18.055567 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":126,"threads_started":1}
W20260806 11:42:18.055800 13629 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:18.050237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:56183 (local address 127.0.113.62:35857)
0806 11:42:18.050409 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:18.050413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:18.050585 (+ 172us) server_negotiation.cc:407] Connection header received
0806 11:42:18.050755 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:18.050759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:18.050813 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:18.050919 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:18.051378 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.051894 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.052702 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.052934 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.053050 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:18.055695 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:56183: BlockingRecv error: recv got EOF from 127.0.113.3:56183 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 11:42:18 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:18 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:18.701494 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:18.696310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:40877)
0806 11:42:18.696615 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:18.696628 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:18.696720 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:18.696995 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:18.696997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:18.697178 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:18.697273 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:18.697278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.698001 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.698003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:18.698544 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:18.698551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.698968 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.698970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:18.699031 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:18.699553 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:18.699564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:18.701367 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
W20260806 11:42:18.701654 13635 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:18.696426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:40877 (local address 127.0.113.62:35857)
0806 11:42:18.696661 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:18.696664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:18.696678 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:42:18.696803 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:18.696805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:18.696873 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:18.696953 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:18.697370 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.697910 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.698638 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.698871 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.698944 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:18.701523 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:40877: BlockingRecv error: recv got EOF from 127.0.113.1:40877 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
Aug 06 11:42:18 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:18 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:18.884029 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:18.878320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:39521)
0806 11:42:18.878634 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:18.878650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:18.878744 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:18.878989 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:18.878991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:18.879199 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:18.879294 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:18.879299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.880062 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.880064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:18.880787 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:18.880798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.881210 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.881213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:18.881270 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:18.881982 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:18.881994 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:18.883912 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
W20260806 11:42:18.884249 13635 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:18.878498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:39521 (local address 127.0.113.62:35857)
0806 11:42:18.878661 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:18.878664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:18.878700 (+ 36us) server_negotiation.cc:407] Connection header received
0806 11:42:18.878832 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:18.878835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:18.878881 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:18.878951 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:18.879427 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.879961 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.880894 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:18.881114 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:18.881195 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:18.884121 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:39521: BlockingRecv error: recv got EOF from 127.0.113.2:39521 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 11:42:19 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:19 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:19.062471 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:19.056615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:46489)
0806 11:42:19.056937 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:19.056951 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:19.057091 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:19.057429 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:19.057432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:19.057731 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:19.057908 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:19.057921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.058835 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.058837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:19.059423 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:19.059431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.059852 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.059854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:19.059917 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:19.060496 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:19.060507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:19.062324 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
W20260806 11:42:19.062597 13635 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:19.056702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:46489 (local address 127.0.113.62:35857)
0806 11:42:19.056834 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:19.056838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:19.057053 (+ 215us) server_negotiation.cc:407] Connection header received
0806 11:42:19.057185 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:19.057187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:19.057265 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:19.057374 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:19.058038 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.058714 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.059526 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.059740 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.059816 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:19.062479 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:46489: BlockingRecv error: recv got EOF from 127.0.113.3:46489 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 11:42:19 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:19 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:19.707650 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:19.702403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:33217)
0806 11:42:19.702681 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:19.702695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:19.702790 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:19.703097 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:19.703100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:19.703288 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:19.703389 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:19.703394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.704234 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.704237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:19.704842 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:19.704851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.705268 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.705270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:19.705329 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:19.705820 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:19.705831 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:19.707537 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
W20260806 11:42:19.707834 13642 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:19.702455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:33217 (local address 127.0.113.62:35857)
0806 11:42:19.702727 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:19.702730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:19.702743 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:42:19.702899 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:19.702903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:19.702979 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:19.703055 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:19.703494 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.704140 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.704955 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.705172 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.705251 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:19.707664 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:33217: BlockingRecv error: recv got EOF from 127.0.113.1:33217 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
Aug 06 11:42:19 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:19 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:19.890661 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:19.884837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:36115)
0806 11:42:19.885134 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:19.885146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:19.885279 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:19.885575 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:19.885578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:19.885817 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:19.885926 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:19.885932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.886759 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.886764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:19.887338 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:19.887346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.887836 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.887840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:19.887960 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:19.888569 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:19.888580 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:19.890489 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":92,"threads_started":1}
W20260806 11:42:19.890722 13642 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:19.884892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:36115 (local address 127.0.113.62:35857)
0806 11:42:19.885060 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:19.885065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:19.885218 (+ 153us) server_negotiation.cc:407] Connection header received
0806 11:42:19.885400 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:19.885403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:19.885457 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:19.885546 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:19.886056 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.886630 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.887472 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:19.887720 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:19.887830 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:19.890614 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:36115: BlockingRecv error: recv got EOF from 127.0.113.2:36115 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 11:42:20 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:20 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:20.069101 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:20.063462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:34171)
0806 11:42:20.063796 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:20.063812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:20.063963 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:20.064287 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:20.064290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:20.064529 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:20.064629 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:20.064635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.065380 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.065382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:20.065924 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:20.065931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.066398 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.066400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:20.066459 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:20.067024 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:20.067038 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:20.068987 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":85,"threads_started":1}
W20260806 11:42:20.069360 13642 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:20.063459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:34171 (local address 127.0.113.62:35857)
0806 11:42:20.063619 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:20.063624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:20.063912 (+ 288us) server_negotiation.cc:407] Connection header received
0806 11:42:20.064080 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:20.064083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:20.064146 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:20.064236 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:20.064734 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.065273 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.066056 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.066272 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.066372 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:20.069118 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:34171: BlockingRecv error: recv got EOF from 127.0.113.3:34171 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 11:42:20 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:20 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:20.714700 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:20.708432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:47193)
0806 11:42:20.708755 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:20.708770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:20.708880 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:20.709124 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:20.709126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:20.709323 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:20.709424 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:20.709430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.710210 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.710216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:20.711069 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:20.711080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.711590 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.711596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:20.711692 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:20.712391 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:20.712404 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:20.714513 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
W20260806 11:42:20.714795 13650 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:20.708488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:47193 (local address 127.0.113.62:35857)
0806 11:42:20.708743 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:20.708746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:20.708840 (+ 94us) server_negotiation.cc:407] Connection header received
0806 11:42:20.708959 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:20.708962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:20.709005 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:20.709085 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:20.709553 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.710090 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.711198 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.711487 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.711571 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:20.714642 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:47193: BlockingRecv error: recv got EOF from 127.0.113.1:47193 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
Aug 06 11:42:20 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:20 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:20.898031 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:20.891568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:40577)
0806 11:42:20.891929 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:20.891943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:20.892083 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:20.892464 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:20.892469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:20.892693 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:20.892814 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:20.892823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.893890 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.893896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:20.894603 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:20.894614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.895183 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.895187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:20.895259 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:20.895778 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:20.895791 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:20.897853 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":105,"threads_started":1}
W20260806 11:42:20.898277 13650 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:20.891682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:40577 (local address 127.0.113.62:35857)
0806 11:42:20.891831 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:20.891836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:20.892019 (+ 183us) server_negotiation.cc:407] Connection header received
0806 11:42:20.892232 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:20.892235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:20.892293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:20.892416 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:20.892936 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.893744 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.894716 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:20.895040 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:20.895139 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:20.898052 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:40577: BlockingRecv error: recv got EOF from 127.0.113.2:40577 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 11:42:21 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:21 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:21.075619 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:21.069889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:48887)
0806 11:42:21.070251 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:21.070264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:21.070363 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:21.070594 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:21.070596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:21.070795 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:21.070907 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:21.070912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.071615 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.071618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:21.072150 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:21.072157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.072564 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.072567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:21.072628 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:21.073209 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:21.073221 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:21.075434 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":134,"threads_started":1}
W20260806 11:42:21.075670 13650 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:21.069956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:48887 (local address 127.0.113.62:35857)
0806 11:42:21.070090 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:21.070094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:21.070330 (+ 236us) server_negotiation.cc:407] Connection header received
0806 11:42:21.070442 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:21.070445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:21.070492 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:21.070554 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:21.071023 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.071528 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.072248 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.072474 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.072548 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:21.075555 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:48887: BlockingRecv error: recv got EOF from 127.0.113.3:48887 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 11:42:21 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:21 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:21.721225 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:21.715513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:35345)
0806 11:42:21.715860 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:21.715876 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:21.716001 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:21.716317 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:21.716321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:21.716533 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:21.716628 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:21.716634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.717429 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.717435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:21.717997 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:21.718005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.718482 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.718485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:21.718571 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:21.719085 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:21.719097 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:21.721057 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":135,"threads_started":1}
W20260806 11:42:21.721294 13655 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:21.715688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:35345 (local address 127.0.113.62:35857)
0806 11:42:21.715921 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:21.715926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:21.715942 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:42:21.716120 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:21.716123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:21.716169 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:21.716271 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:21.716751 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.717306 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.718136 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.718368 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.718477 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:21.721174 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:35345: BlockingRecv error: recv got EOF from 127.0.113.1:35345 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":61,"threads_started":1}
Aug 06 11:42:21 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:21 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:21.904382 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:21.898859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:34741)
0806 11:42:21.899120 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:21.899136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:21.899243 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:21.899481 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:21.899483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:21.899671 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:21.899770 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:21.899775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.900651 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.900654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:21.901200 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:21.901207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.901605 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.901608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:21.901668 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:21.902172 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:21.902182 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:21.904166 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
W20260806 11:42:21.904462 13655 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:21.898897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:34741 (local address 127.0.113.62:35857)
0806 11:42:21.899061 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:21.899065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:21.899216 (+ 151us) server_negotiation.cc:407] Connection header received
0806 11:42:21.899323 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:21.899325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:21.899370 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:21.899438 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:21.899886 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.900563 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.901299 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:21.901510 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:21.901587 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:21.904309 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:34741: BlockingRecv error: recv got EOF from 127.0.113.2:34741 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 11:42:22 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:22 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:22.082013 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:22.076490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:53909)
0806 11:42:22.076776 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:22.076797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:22.076896 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:22.077168 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:22.077171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:22.077380 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:22.077479 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:22.077484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.078177 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.078179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:22.078720 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:22.078727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.079125 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.079128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:22.079207 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:22.079807 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:22.079820 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:22.081842 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
W20260806 11:42:22.082255 13655 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:22.076642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:53909 (local address 127.0.113.62:35857)
0806 11:42:22.076869 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:22.076872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:22.076887 (+ 15us) server_negotiation.cc:407] Connection header received
0806 11:42:22.076997 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:22.077000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:22.077048 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:22.077119 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:22.077584 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.078086 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.078827 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.079028 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.079104 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:22.082120 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:53909: BlockingRecv error: recv got EOF from 127.0.113.3:53909 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 06 11:42:22 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:22 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:22.732970 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:22.722051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:49197)
0806 11:42:22.722351 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:22.722379 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:22.722482 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:22.722707 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:22.722709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:22.722892 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:22.722984 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:22.722988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.723707 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.723709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:22.724243 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:22.724249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.724696 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.724699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:22.724757 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:22.730530 (+ 5773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:22.730547 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:22.732811 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":97,"threads_started":1}
W20260806 11:42:22.733096 13660 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:22.722115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:49197 (local address 127.0.113.62:35857)
0806 11:42:22.722380 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:22.722383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:22.722437 (+ 54us) server_negotiation.cc:407] Connection header received
0806 11:42:22.722546 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:22.722548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:22.722591 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:22.722666 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:22.723093 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.723616 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.724369 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.724596 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.724676 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:22.732965 (+ 8289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:49197: BlockingRecv error: recv got EOF from 127.0.113.1:49197 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":63,"threads_started":1}
Aug 06 11:42:22 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:22 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:22.910933 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:22.905237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:39785)
0806 11:42:22.905569 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:22.905583 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:22.905722 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:22.906007 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:22.906010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:22.906225 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:22.906323 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:22.906329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.907106 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.907112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:22.907656 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:22.907664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.908161 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.908165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:22.908264 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:22.908867 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:22.908880 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:22.910732 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":118,"threads_started":1}
W20260806 11:42:22.910984 13660 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:22.905307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:39785 (local address 127.0.113.62:35857)
0806 11:42:22.905481 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:22.905486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:22.905662 (+ 176us) server_negotiation.cc:407] Connection header received
0806 11:42:22.905823 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:22.905827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:22.905885 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:22.905978 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:22.906453 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.906981 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.907799 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:22.908032 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:22.908144 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:22.910861 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:39785: BlockingRecv error: recv got EOF from 127.0.113.2:39785 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 11:42:23 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:23 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:23.088443 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:23.082752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:45645)
0806 11:42:23.083059 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:23.083073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:23.083231 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:23.083469 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:23.083471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:23.083667 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:23.083842 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:23.083848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.084655 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.084657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:23.085266 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:23.085274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.085748 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.085752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:23.085836 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:23.086432 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:23.086444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:23.088235 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
W20260806 11:42:23.088495 13660 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:23.082918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:45645 (local address 127.0.113.62:35857)
0806 11:42:23.083080 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:23.083083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:23.083162 (+ 79us) server_negotiation.cc:407] Connection header received
0806 11:42:23.083314 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:23.083316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:23.083363 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:23.083427 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:23.083952 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.084558 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.085387 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.085628 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.085744 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:23.088389 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:45645: BlockingRecv error: recv got EOF from 127.0.113.3:45645 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 11:42:23 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:23 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:23.739006 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:23.733744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:47807)
0806 11:42:23.734060 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:23.734078 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:23.734185 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:23.734414 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:23.734416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:23.734603 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:23.734702 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:23.734707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.735433 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.735436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:23.735989 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:23.735996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.736430 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.736433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:23.736497 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:23.737088 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:23.737100 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:23.738889 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
W20260806 11:42:23.739184 13666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:23.733941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:47807 (local address 127.0.113.62:35857)
0806 11:42:23.734167 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:23.734171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:23.734184 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:42:23.734254 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:23.734256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:23.734297 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:23.734371 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:23.734806 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.735338 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.736089 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.736324 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.736399 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:23.739036 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:47807: BlockingRecv error: recv got EOF from 127.0.113.1:47807 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
Aug 06 11:42:23 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:23 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:23.917451 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:23.911835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:45503)
0806 11:42:23.912150 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:23.912164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:23.912272 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:23.912540 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:23.912543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:23.912729 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:23.912843 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:23.912848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.913576 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.913578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:23.914120 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:23.914127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.914693 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.914697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:23.914789 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:23.915388 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:23.915400 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:23.917337 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":119,"threads_started":1}
W20260806 11:42:23.917635 13666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:23.911935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:45503 (local address 127.0.113.62:35857)
0806 11:42:23.912082 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:23.912085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:23.912247 (+ 162us) server_negotiation.cc:407] Connection header received
0806 11:42:23.912385 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:23.912387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:23.912431 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:23.912500 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:23.912950 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.913483 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.914220 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:23.914542 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:23.914681 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:23.917491 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:45503: BlockingRecv error: recv got EOF from 127.0.113.2:45503 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 11:42:24 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:24 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:24.095487 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:24.089262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:47667)
0806 11:42:24.089607 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:24.089626 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:24.089804 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:24.090115 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:24.090118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:24.090360 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:24.090472 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:24.090478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.091319 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.091325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:24.091935 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:24.091944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.092516 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.092521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:24.092615 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:24.093246 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:24.093259 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:24.095305 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":124,"threads_started":1}
W20260806 11:42:24.095562 13666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:24.089443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:47667 (local address 127.0.113.62:35857)
0806 11:42:24.089608 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:24.089614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:24.089716 (+ 102us) server_negotiation.cc:407] Connection header received
0806 11:42:24.089921 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:24.089926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:24.089987 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:24.090085 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:24.090609 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.091190 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.092070 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.092391 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.092511 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:24.095445 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:47667: BlockingRecv error: recv got EOF from 127.0.113.3:47667 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 11:42:24 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:24 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:24.745307 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:24.739792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:40993)
0806 11:42:24.740086 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:24.740100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:24.740191 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:24.740496 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:24.740499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:24.740688 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:24.740790 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:24.740795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.741590 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.741593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:24.742201 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:24.742209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.742661 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.742664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:24.742724 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:24.743241 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:24.743252 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:24.745191 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
W20260806 11:42:24.745519 13671 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:24.739911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:40993 (local address 127.0.113.62:35857)
0806 11:42:24.740164 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:24.740168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:24.740181 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:42:24.740259 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:24.740262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:24.740341 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:24.740426 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:24.740937 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.741488 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.742316 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.742556 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.742638 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:24.745360 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:40993: BlockingRecv error: recv got EOF from 127.0.113.1:40993 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
Aug 06 11:42:24 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:24 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:24.923969 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:24.918256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:47023)
0806 11:42:24.918533 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:24.918555 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:24.918654 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:24.918885 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:24.918887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:24.919072 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:24.919230 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:24.919237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.920112 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.920116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:24.920804 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:24.920815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.921276 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.921280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:24.921372 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:24.921967 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:24.921980 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:24.923784 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":115,"threads_started":1}
W20260806 11:42:24.924088 13671 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:24.918335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:47023 (local address 127.0.113.62:35857)
0806 11:42:24.918472 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:24.918475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:24.918622 (+ 147us) server_negotiation.cc:407] Connection header received
0806 11:42:24.918729 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:24.918731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:24.918775 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:24.918845 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:24.919366 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.919993 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.920930 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:24.921159 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:24.921270 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:24.923970 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:47023: BlockingRecv error: recv got EOF from 127.0.113.2:47023 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 11:42:25 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:25 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:25.101440 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:25.096226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:46113)
0806 11:42:25.096590 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:25.096602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:25.096700 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:25.096939 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:25.096941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:25.097117 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:25.097208 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:25.097212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.097860 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.097862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:25.098369 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:25.098376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.098765 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.098767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:25.098853 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:25.099432 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:25.099445 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:25.101324 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":153,"threads_started":1}
W20260806 11:42:25.101560 13671 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:25.096374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:46113 (local address 127.0.113.62:35857)
0806 11:42:25.096541 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:25.096544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:25.096670 (+ 126us) server_negotiation.cc:407] Connection header received
0806 11:42:25.096774 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:25.096776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:25.096820 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:25.096899 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:25.097298 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.097765 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.098461 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.098654 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.098723 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:25.101450 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:46113: BlockingRecv error: recv got EOF from 127.0.113.3:46113 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 11:42:25 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:25 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:25.752218 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:25.746227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:58235)
0806 11:42:25.746576 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:25.746590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:25.746691 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:25.747008 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:25.747013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:25.747267 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:25.747425 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:25.747438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.748560 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.748563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:25.749109 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:25.749117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.749572 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.749575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:25.749638 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:25.750134 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:25.750145 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:25.752099 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
W20260806 11:42:25.752478 13676 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:25.746282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:58235 (local address 127.0.113.62:35857)
0806 11:42:25.746546 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:25.746550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:25.746663 (+ 113us) server_negotiation.cc:407] Connection header received
0806 11:42:25.746794 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:25.746797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:25.746845 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:25.746939 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:25.747570 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.748451 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.749217 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.749470 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.749554 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:25.752261 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:58235: BlockingRecv error: recv got EOF from 127.0.113.1:58235 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
Aug 06 11:42:25 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:25 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:25.930054 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:25.924757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:59707)
0806 11:42:25.925121 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:25.925136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:25.925236 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:25.925477 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:25.925479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:25.925668 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:25.925764 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:25.925769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.926480 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.926482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:25.927012 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:25.927019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.927432 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.927435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:25.927494 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:25.928031 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:25.928042 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:25.929940 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":150,"threads_started":1}
W20260806 11:42:25.930197 13676 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:25.924863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:59707 (local address 127.0.113.62:35857)
0806 11:42:25.925008 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:25.925011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:25.925204 (+ 193us) server_negotiation.cc:407] Connection header received
0806 11:42:25.925314 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:25.925317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:25.925363 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:25.925435 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:25.925885 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.926388 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.927110 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:25.927315 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:25.927390 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:25.930074 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:59707: BlockingRecv error: recv got EOF from 127.0.113.2:59707 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 11:42:26 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:26 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:26.107906 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:26.102345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:51685)
0806 11:42:26.102622 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:26.102636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:26.102738 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:26.102996 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:26.102999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:26.103267 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:26.103392 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:26.103400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.104153 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.104159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:26.104812 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:26.104822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.105298 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.105302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:26.105388 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:26.105909 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:26.105921 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:26.107788 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":100,"threads_started":1}
W20260806 11:42:26.108110 13676 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:26.102395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:51685 (local address 127.0.113.62:35857)
0806 11:42:26.102557 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:26.102562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:26.102706 (+ 144us) server_negotiation.cc:407] Connection header received
0806 11:42:26.102817 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:26.102819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:26.102871 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:26.102968 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:26.103520 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.104035 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.104950 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.105173 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.105286 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:26.107941 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:51685: BlockingRecv error: recv got EOF from 127.0.113.3:51685 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 11:42:26 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:26 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:26.758513 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:26.753075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:58321)
0806 11:42:26.753390 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:26.753405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:26.753520 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:26.753812 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:26.753815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:26.754008 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:26.754109 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:26.754123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.754877 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.754880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:26.755417 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:26.755425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.755850 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.755853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:26.755921 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:26.756506 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:26.756518 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:26.758379 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
W20260806 11:42:26.758718 13682 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:26.753133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:58321 (local address 127.0.113.62:35857)
0806 11:42:26.753388 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:26.753393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:26.753490 (+ 97us) server_negotiation.cc:407] Connection header received
0806 11:42:26.753602 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:26.753605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:26.753648 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:26.753727 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:26.754233 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.754774 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.755519 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.755728 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.755809 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:26.758539 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:58321: BlockingRecv error: recv got EOF from 127.0.113.1:58321 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
Aug 06 11:42:26 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:26 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:26.937060 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:26.931038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:37631)
0806 11:42:26.931330 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:26.931346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:26.931449 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:26.931740 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:26.931743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:26.931945 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:26.932047 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:26.932053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.932980 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.932985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:26.933565 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:26.933574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.934097 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.934102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:26.934202 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:26.934752 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:26.934764 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:26.936864 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":101,"threads_started":1}
W20260806 11:42:26.937100 13682 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:26.931219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:37631 (local address 127.0.113.62:35857)
0806 11:42:26.931362 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:26.931366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:26.931397 (+ 31us) server_negotiation.cc:407] Connection header received
0806 11:42:26.931533 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:26.931535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:26.931581 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:26.931654 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:26.932153 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.932860 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.933710 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:26.933961 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:26.934070 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:26.936995 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:37631: BlockingRecv error: recv got EOF from 127.0.113.2:37631 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 11:42:27 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:27 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:27.114372 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:27.108695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:51125)
0806 11:42:27.108975 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:27.108987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:27.109115 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:27.109403 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:27.109406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:27.109651 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:27.109749 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:27.109755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.110560 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.110565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:27.111146 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:27.111154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.111638 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.111642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:27.111723 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:27.112292 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:27.112353 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:27.114235 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
W20260806 11:42:27.114531 13682 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:27.108745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:51125 (local address 127.0.113.62:35857)
0806 11:42:27.108911 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:27.108916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:27.109057 (+ 141us) server_negotiation.cc:407] Connection header received
0806 11:42:27.109219 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:27.109222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:27.109273 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:27.109363 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:27.109890 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.110440 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.111278 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.111523 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.111633 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:27.114374 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:51125: BlockingRecv error: recv got EOF from 127.0.113.3:51125 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 11:42:27 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:27 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:27.771581 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:27.759307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:60595)
0806 11:42:27.759594 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:27.759608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:27.759702 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:27.760001 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:27.760003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:27.760207 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:27.760334 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:27.760341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.761168 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.761171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:27.761737 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:27.761745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.762310 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.762313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:27.762389 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:27.769215 (+ 6826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:27.769229 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:27.771405 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
W20260806 11:42:27.771860 13687 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:27.759372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:60595 (local address 127.0.113.62:35857)
0806 11:42:27.759673 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:27.759676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:27.759689 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:42:27.759825 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:27.759829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:27.759879 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:27.759964 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:27.760485 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.761033 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.761928 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.762164 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.762255 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:27.771559 (+ 9304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:60595: BlockingRecv error: recv got EOF from 127.0.113.1:60595 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":61,"threads_started":1}
Aug 06 11:42:27 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:27 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:27.943604 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:27.937904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:50585)
0806 11:42:27.938209 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:27.938222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:27.938333 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:27.938576 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:27.938578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:27.938762 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:27.938857 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:27.938862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.939564 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.939567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:27.940143 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:27.940151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.940642 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.940646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:27.940715 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:27.941274 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:27.941292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:27.943479 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":145,"threads_started":1}
W20260806 11:42:27.943779 13687 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:27.938011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:50585 (local address 127.0.113.62:35857)
0806 11:42:27.938196 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:27.938203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:27.938299 (+ 96us) server_negotiation.cc:407] Connection header received
0806 11:42:27.938416 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:27.938418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:27.938463 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:27.938536 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:27.938959 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.939471 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.940249 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:27.940527 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:27.940608 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:27.943622 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:50585: BlockingRecv error: recv got EOF from 127.0.113.2:50585 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 11:42:28 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:28 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.3@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:28.121474 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:28.115273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.3:38923)
0806 11:42:28.115647 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:28.115663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:28.115816 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:28.116156 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:28.116159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:28.116498 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:28.116630 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:28.116640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.117415 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.117420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:28.117984 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:28.117992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.118472 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.118475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:28.118575 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:28.119190 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:28.119212 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:28.121354 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":100,"threads_started":1}
W20260806 11:42:28.121639 13687 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:28.115425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:38923 (local address 127.0.113.62:35857)
0806 11:42:28.115624 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:28.115630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:28.115753 (+ 123us) server_negotiation.cc:407] Connection header received
0806 11:42:28.115967 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:28.115984 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:28.116036 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:28.116126 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:28.116760 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.117296 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.118110 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.118351 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.118465 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:28.121509 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.3:38923: BlockingRecv error: recv got EOF from 127.0.113.3:38923 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 11:42:28 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:28 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.1@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:28.778138 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:28.772572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.1:51931)
0806 11:42:28.772861 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:28.772875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:28.772961 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:28.773272 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:28.773275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:28.773474 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:28.773591 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:28.773597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.774408 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.774412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:28.774990 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:28.775000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.775433 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.775436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:28.775498 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:28.776010 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:28.776022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:28.777992 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
W20260806 11:42:28.778422 13692 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:28.772788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:51931 (local address 127.0.113.62:35857)
0806 11:42:28.773038 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:28.773042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:28.773053 (+ 11us) server_negotiation.cc:407] Connection header received
0806 11:42:28.773088 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:28.773090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:28.773132 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:28.773215 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:28.773712 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.774271 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.775107 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.775325 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.775407 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:28.778264 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.1:51931: BlockingRecv error: recv got EOF from 127.0.113.1:51931 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
Aug 06 11:42:28 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 11:42:28 dist-test-slave-45dx krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.113.2@KRBTEST.COM for oryx/127.0.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 11:42:28.950338 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:28.944580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:35857 (local address 127.0.113.2:42401)
0806 11:42:28.944880 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:28.944893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:28.944992 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:28.945229 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:28.945231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:28.945420 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:28.945548 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:28.945553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.946479 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.946483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:28.947051 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:28.947059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.947538 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.947541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:28.947605 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:28.948189 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:28.948202 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:28.950157 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.113.62:35857: Server oryx/127.0.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":97,"threads_started":1}
W20260806 11:42:28.950393 13692 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:28.944712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:42401 (local address 127.0.113.62:35857)
0806 11:42:28.944856 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:28.944860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:28.944961 (+ 101us) server_negotiation.cc:407] Connection header received
0806 11:42:28.945072 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:28.945074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:28.945117 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:28.945184 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:28.945694 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.946354 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.947188 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:28.947434 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:28.947520 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:28.950287 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.113.2:42401: BlockingRecv error: recv got EOF from 127.0.113.2:42401 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:42:29.010816 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 13027
I20260806 11:42:29.022571 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 13241
I20260806 11:42:29.030869 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 13421
I20260806 11:42:29.042493 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 12932
2026-08-06T11:42:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18252 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T11:42:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:42:29Z Disabled control of system clock
I20260806 11:42:29.081717 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:40085
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:40085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 11:42:29.193565 13698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:29.193823 13698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:29.193856 13698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:29.197831 13698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:29.197885 13698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:29.197904 13698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:29.197921 13698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:29.197935 13698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:29.202541 13698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:40085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:40085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:29.203548 13698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:29.204906 13698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:29.213868 13705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:29.213954 13698 server_base.cc:1034] running on GCE node
W20260806 11:42:29.213865 13706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:29.213865 13708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:29.214788 13698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:29.215947 13698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:29.217137 13698 hybrid_clock.cc:648] HybridClock initialized: now 1786016549217086 us; error 64 us; skew 500 ppm
I20260806 11:42:29.219126 13698 webserver.cc:460] Webserver started at http://127.0.113.62:40681/ using document root <none> and password file <none>
I20260806 11:42:29.219750 13698 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:29.219851 13698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:29.220116 13698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:29.222323 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "9587828724a241a8a72c50d6f6c0eb65"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "d1c89d15eed250fb1ec1c6b441e27f46"
server_key_iv: "c0bbba080f246e65a1546c3c27308036"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.222930 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "9587828724a241a8a72c50d6f6c0eb65"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "d1c89d15eed250fb1ec1c6b441e27f46"
server_key_iv: "c0bbba080f246e65a1546c3c27308036"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.226622 13698 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 11:42:29.229031 13714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.230316 13698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:42:29.230471 13698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "9587828724a241a8a72c50d6f6c0eb65"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "d1c89d15eed250fb1ec1c6b441e27f46"
server_key_iv: "c0bbba080f246e65a1546c3c27308036"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.230590 13698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:29.254540 13698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:29.255290 13698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:29.255518 13698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:29.262884 13698 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:40085
I20260806 11:42:29.262895 13799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:40085 every 8 connection(s)
I20260806 11:42:29.263851 13698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:42:29.267686 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:29.267968 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 13698
I20260806 11:42:29.268091 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:42:29.268463 452 external_mini_cluster.cc:1428] Setting key fbe2b73fc4f87ad134ebec9e6bc8556c
I20260806 11:42:29.273995 13800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65: Bootstrap starting.
I20260806 11:42:29.276536 13800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:29.277287 13800 log.cc:824] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:29.277743 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:29.269909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.0.113.62:40085)
0806 11:42:29.270527 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:29.270543 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:29.270578 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:42:29.271229 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:29.271244 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:29.271637 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:29.271904 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:29.272263 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.273253 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.274435 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.274897 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.275016 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:29.275909 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:29.275915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:29.275926 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:42:29.276283 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:29.276395 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:42:29.276417 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:29.276492 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:29.276696 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 11:42:29.276975 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
I20260806 11:42:29.279345 13800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65: No bootstrap required, opened a new log
I20260806 11:42:29.283295 13800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9587828724a241a8a72c50d6f6c0eb65" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40085 } }
I20260806 11:42:29.283504 13800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:29.283538 13800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9587828724a241a8a72c50d6f6c0eb65, State: Initialized, Role: FOLLOWER
I20260806 11:42:29.283861 13800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9587828724a241a8a72c50d6f6c0eb65" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40085 } }
I20260806 11:42:29.283964 13800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:42:29.284000 13800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:42:29.284138 13800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:29.285020 13800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9587828724a241a8a72c50d6f6c0eb65" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40085 } }
I20260806 11:42:29.285395 13800 leader_election.cc:302] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9587828724a241a8a72c50d6f6c0eb65; no voters:
I20260806 11:42:29.285771 13800 leader_election.cc:288] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:42:29.285840 13806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:29.286044 13806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [term 1 LEADER]: Becoming Leader. State: Replica: 9587828724a241a8a72c50d6f6c0eb65, State: Running, Role: LEADER
I20260806 11:42:29.286413 13806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9587828724a241a8a72c50d6f6c0eb65" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40085 } }
I20260806 11:42:29.287024 13800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:42:29.288089 13808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9587828724a241a8a72c50d6f6c0eb65. Latest consensus state: current_term: 1 leader_uuid: "9587828724a241a8a72c50d6f6c0eb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9587828724a241a8a72c50d6f6c0eb65" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40085 } } }
I20260806 11:42:29.288224 13808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:29.288516 13807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9587828724a241a8a72c50d6f6c0eb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9587828724a241a8a72c50d6f6c0eb65" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40085 } } }
I20260806 11:42:29.288612 13807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:29.289386 13816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:42:29.292371 13816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:42:29.297537 13816 catalog_manager.cc:1347] Generated new cluster ID: eaf87b61f1e44020904c48139835e47a
I20260806 11:42:29.297613 13816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:42:29.316915 13816 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:42:29.317771 13816 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:42:29.340202 13816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9587828724a241a8a72c50d6f6c0eb65: Generated new TSK 0
I20260806 11:42:29.341130 13816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:42:29.357312 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40085
--builtin_ntp_servers=127.0.113.20:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 11:42:29.468803 13831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:29.469090 13831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:29.469156 13831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:29.473177 13831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:29.473246 13831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:29.473328 13831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:42:29.477831 13831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:29.478864 13831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:29.480386 13831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 11:42:29.490389 13831 server_base.cc:1034] running on GCE node
W20260806 11:42:29.490351 13840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:29.490353 13841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:29.490345 13844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:29.491041 13831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:29.491621 13831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:29.492784 13831 hybrid_clock.cc:648] HybridClock initialized: now 1786016549492769 us; error 38 us; skew 500 ppm
I20260806 11:42:29.494621 13831 webserver.cc:460] Webserver started at http://127.0.113.1:34061/ using document root <none> and password file <none>
I20260806 11:42:29.495175 13831 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:29.495236 13831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:29.495396 13831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:29.497449 13831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "412a349bcbcc465d9c76d5e0995b267d"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "cfa69451657fc95efdda0f9f484e82c6"
server_key_iv: "942ad85fe1afddd3c3130e00a05de301"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.497979 13831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "412a349bcbcc465d9c76d5e0995b267d"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "cfa69451657fc95efdda0f9f484e82c6"
server_key_iv: "942ad85fe1afddd3c3130e00a05de301"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.501531 13831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:42:29.504062 13851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.505290 13831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:29.505409 13831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "412a349bcbcc465d9c76d5e0995b267d"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "cfa69451657fc95efdda0f9f484e82c6"
server_key_iv: "942ad85fe1afddd3c3130e00a05de301"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.505523 13831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:29.526638 13831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:29.527349 13831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:29.527566 13831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:29.528165 13831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:29.529122 13831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:29.529168 13831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.529247 13831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:29.529277 13831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.538738 13831 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:36247
I20260806 11:42:29.538763 14013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:36247 every 8 connection(s)
I20260806 11:42:29.539647 13831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:42:29.544772 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 13831
I20260806 11:42:29.544914 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:42:29.545239 452 external_mini_cluster.cc:1428] Setting key e58cbe7b4f55e374d7f025b56264a8ec
I20260806 11:42:29.547204 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40085
--builtin_ntp_servers=127.0.113.20:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 11:42:29.549438 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:29.541526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:45713 (local address 127.0.113.62:40085)
0806 11:42:29.541674 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:29.541679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:29.542751 (+ 1072us) server_negotiation.cc:407] Connection header received
0806 11:42:29.544080 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:29.544082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:29.544133 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:29.544210 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:29.545383 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.546061 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.547419 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.547632 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.547746 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:29.548920 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:29.548924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:29.548926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:42:29.548939 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:29.548974 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:42:29.548982 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:29.549060 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:29.549220 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 11:42:29.549288 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:42:29.550299 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:29.541979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40085 (local address 127.0.113.1:45713)
0806 11:42:29.542606 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:29.542646 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:29.543897 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:29.544397 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:29.544403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:29.544525 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:29.545216 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:29.545236 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.546202 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.546205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:29.547191 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:29.547202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.547788 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.547792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:29.547952 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:29.548660 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:42:29.548662 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:29.548704 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:42:29.548767 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:29.549076 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:29.549090 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:29.549096 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 11:42:29.549219 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 11:42:29.549541 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20260806 11:42:29.551570 14014 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40085
I20260806 11:42:29.551803 14014 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:29.552402 14014 heartbeater.cc:511] Master 127.0.113.62:40085 requested a full tablet report, sending...
I20260806 11:42:29.553918 13738 ts_manager.cc:194] Registered new tserver with Master: 412a349bcbcc465d9c76d5e0995b267d (127.0.113.1:36247)
I20260806 11:42:29.555042 13738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.1:45713
W20260806 11:42:29.659677 14017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:29.659952 14017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:29.660022 14017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:29.664125 14017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:29.664202 14017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:29.664285 14017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:42:29.669122 14017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:29.670147 14017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:29.671478 14017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:29.680991 14022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:29.681034 14017 server_base.cc:1034] running on GCE node
W20260806 11:42:29.680997 14023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:29.681008 14025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:29.681650 14017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:29.682272 14017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:29.683431 14017 hybrid_clock.cc:648] HybridClock initialized: now 1786016549683416 us; error 26 us; skew 500 ppm
I20260806 11:42:29.685317 14017 webserver.cc:460] Webserver started at http://127.0.113.2:41753/ using document root <none> and password file <none>
I20260806 11:42:29.685894 14017 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:29.685957 14017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:29.686211 14017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:29.688530 14017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "395ac5cfa38546d7ac6f989508e5df05"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "d29d4cc2fbed13ce2c2d10a92864e85c"
server_key_iv: "ec4e42ac0b2d9e98b520d50c34548680"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.689155 14017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "395ac5cfa38546d7ac6f989508e5df05"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "d29d4cc2fbed13ce2c2d10a92864e85c"
server_key_iv: "ec4e42ac0b2d9e98b520d50c34548680"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.692706 14017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:42:29.695120 14031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.696377 14017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:29.696506 14017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "395ac5cfa38546d7ac6f989508e5df05"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "d29d4cc2fbed13ce2c2d10a92864e85c"
server_key_iv: "ec4e42ac0b2d9e98b520d50c34548680"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.696615 14017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:29.738080 14017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:29.738785 14017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:29.739005 14017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:29.739615 14017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:29.740628 14017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:29.740696 14017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.740779 14017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:29.740829 14017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.749655 14017 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:39527
I20260806 11:42:29.749688 14207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:39527 every 8 connection(s)
I20260806 11:42:29.750572 14017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:42:29.755285 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 14017
I20260806 11:42:29.755441 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:42:29.755745 452 external_mini_cluster.cc:1428] Setting key f8b766e8d1c739e406073a83024ec276
I20260806 11:42:29.757742 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40085
--builtin_ntp_servers=127.0.113.20:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 11:42:29.760430 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:29.752417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:54803 (local address 127.0.113.62:40085)
0806 11:42:29.752616 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:29.752621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:29.753689 (+ 1068us) server_negotiation.cc:407] Connection header received
0806 11:42:29.755094 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:29.755099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:29.755162 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:29.755263 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:29.756278 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.757024 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.758321 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.758549 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.758649 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:29.759924 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:29.759927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:29.759929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:42:29.759941 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:29.759961 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:42:29.759969 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:29.760024 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:29.760205 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 11:42:29.760254 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":74}
I20260806 11:42:29.761587 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:29.752881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40085 (local address 127.0.113.2:54803)
0806 11:42:29.753543 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:29.753582 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:29.754843 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:29.755465 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:29.755490 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:29.755590 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:29.756130 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:29.756144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.757165 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.757174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:29.758189 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:29.758200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.758700 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.758705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:29.758858 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:29.759588 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:42:29.759590 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:29.759655 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:42:29.759741 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:29.760055 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:29.760067 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:29.760074 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 11:42:29.760213 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 11:42:29.760572 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
I20260806 11:42:29.762791 14212 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40085
I20260806 11:42:29.763038 14212 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:29.763515 14212 heartbeater.cc:511] Master 127.0.113.62:40085 requested a full tablet report, sending...
I20260806 11:42:29.764547 13738 ts_manager.cc:194] Registered new tserver with Master: 395ac5cfa38546d7ac6f989508e5df05 (127.0.113.2:39527)
I20260806 11:42:29.765084 13738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.2:54803
W20260806 11:42:29.871639 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:29.871953 14215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:29.872032 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:29.876133 14215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:29.876214 14215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:29.876318 14215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:42:29.881127 14215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:29.882220 14215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:29.883584 14215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:29.893255 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:29.893265 14222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:29.893541 14215 server_base.cc:1034] running on GCE node
W20260806 11:42:29.893267 14221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:29.894088 14215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:29.894690 14215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:29.895887 14215 hybrid_clock.cc:648] HybridClock initialized: now 1786016549895853 us; error 44 us; skew 500 ppm
I20260806 11:42:29.897701 14215 webserver.cc:460] Webserver started at http://127.0.113.3:41417/ using document root <none> and password file <none>
I20260806 11:42:29.898278 14215 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:29.898371 14215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:29.898609 14215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:29.900803 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "e46108f62de643798042c7c9072c2cc9"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "915320947037dc5ea413629b2c369ce0"
server_key_iv: "1def9e6d82d24787cdb862907bc0d79f"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.901378 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "e46108f62de643798042c7c9072c2cc9"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "915320947037dc5ea413629b2c369ce0"
server_key_iv: "1def9e6d82d24787cdb862907bc0d79f"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.904950 14215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:42:29.907392 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.908587 14215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:29.908718 14215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "e46108f62de643798042c7c9072c2cc9"
format_stamp: "Formatted at 2026-08-06 11:42:29 on dist-test-slave-45dx"
server_key: "915320947037dc5ea413629b2c369ce0"
server_key_iv: "1def9e6d82d24787cdb862907bc0d79f"
server_key_version: "encryptionkey@0"
I20260806 11:42:29.908845 14215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:29.946568 14215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:29.947340 14215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:29.947557 14215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:29.948194 14215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:29.949232 14215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:29.949307 14215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.949379 14215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:29.949432 14215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:29.958642 14215 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:43125
I20260806 11:42:29.958681 14387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:43125 every 8 connection(s)
I20260806 11:42:29.959605 14215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:42:29.966121 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 14215
I20260806 11:42:29.966236 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:42:29.966526 452 external_mini_cluster.cc:1428] Setting key bb790abe5a1df6748e3948b1061cb6ca
I20260806 11:42:29.969203 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:29.961451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:50033 (local address 127.0.113.62:40085)
0806 11:42:29.961638 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:29.961643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:29.962711 (+ 1068us) server_negotiation.cc:407] Connection header received
0806 11:42:29.964019 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:29.964022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:29.964068 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:29.964145 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:29.965043 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.965540 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.967036 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.967359 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.967509 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:29.968763 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:29.968773 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:29.968776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:42:29.968786 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:29.968805 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:42:29.968813 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:29.968913 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:29.969051 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 11:42:29.969100 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 11:42:29.969998 14392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:29.961914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40085 (local address 127.0.113.3:50033)
0806 11:42:29.962545 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:29.962586 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:29.963818 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:29.964342 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:29.964353 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:29.964442 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:29.964913 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:29.964926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.965644 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.965646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:29.966876 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:29.966886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:29.967538 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:29.967543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:29.967757 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:29.968514 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:42:29.968516 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:29.968553 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:42:29.968615 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:29.968903 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:29.968915 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:29.968920 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 11:42:29.969028 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 11:42:29.969229 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260806 11:42:29.971045 14388 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40085
I20260806 11:42:29.971287 14388 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:29.971781 14388 heartbeater.cc:511] Master 127.0.113.62:40085 requested a full tablet report, sending...
I20260806 11:42:29.972736 13738 ts_manager.cc:194] Registered new tserver with Master: e46108f62de643798042c7c9072c2cc9 (127.0.113.3:43125)
I20260806 11:42:29.973264 13738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.3:50033
I20260806 11:42:29.980543 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 11:42:29.983363 13804 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:29.982447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.0.113.62:40085)
0806 11:42:29.982591 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:29.982596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:29.982795 (+ 199us) server_negotiation.cc:407] Connection header received
0806 11:42:29.982874 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:29.982876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:29.982917 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:29.982980 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:29.983252 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59388: BlockingRecv error: recv got EOF from 127.0.0.1:59388 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:42:29.983577 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 13831
I20260806 11:42:30.008009 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 14017
I20260806 11:42:30.019157 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 14215
I20260806 11:42:30.029973 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 13698
2026-08-06T11:42:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (983 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T11:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:42:30Z Disabled control of system clock
I20260806 11:42:30.064703 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33099
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:33099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 11:42:30.173075 14404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:30.173317 14404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:30.173352 14404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:30.177233 14404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:30.177289 14404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:30.177304 14404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:30.177320 14404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:30.177335 14404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:30.181911 14404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:33099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:30.182858 14404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:30.184145 14404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:30.192631 14413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:30.192714 14404 server_base.cc:1034] running on GCE node
W20260806 11:42:30.192642 14409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:30.192677 14410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:30.193501 14404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:30.194545 14404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:30.195762 14404 hybrid_clock.cc:648] HybridClock initialized: now 1786016550195749 us; error 35 us; skew 500 ppm
I20260806 11:42:30.197723 14404 webserver.cc:460] Webserver started at http://127.0.113.62:43409/ using document root <none> and password file <none>
I20260806 11:42:30.198290 14404 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:30.198393 14404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:30.198628 14404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:30.200798 14404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "8daa90832d714aecbdd8d6eede8b0f86"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "0187c90dbf4e705628b65a9c1f3e26a5"
server_key_iv: "0a1a13547d37d48861c807fc3ec63abf"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.201388 14404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "8daa90832d714aecbdd8d6eede8b0f86"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "0187c90dbf4e705628b65a9c1f3e26a5"
server_key_iv: "0a1a13547d37d48861c807fc3ec63abf"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.205242 14404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 11:42:30.207702 14427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.209040 14404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:42:30.209180 14404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "8daa90832d714aecbdd8d6eede8b0f86"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "0187c90dbf4e705628b65a9c1f3e26a5"
server_key_iv: "0a1a13547d37d48861c807fc3ec63abf"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.209292 14404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:30.213828 14404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:30.214418 14404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:30.214593 14404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:30.221375 14404 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:33099
I20260806 11:42:30.221403 14504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:33099 every 8 connection(s)
I20260806 11:42:30.222262 14404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:42:30.225406 14505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:30.230036 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 14404
I20260806 11:42:30.230163 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:42:30.230499 452 external_mini_cluster.cc:1428] Setting key 2bade32795645a7c029c70b635140c8f
I20260806 11:42:30.231120 14505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86: Bootstrap starting.
I20260806 11:42:30.233558 14505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:30.234347 14505 log.cc:824] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:30.236040 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:30.231970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.0.113.62:33099)
0806 11:42:30.232624 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:30.232640 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:30.232678 (+ 38us) server_negotiation.cc:407] Connection header received
0806 11:42:30.233271 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:30.233286 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:30.233633 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:30.233999 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:30.234807 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:30.234811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:30.234822 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:42:30.234832 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:30.234891 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:42:30.234907 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:30.235031 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:30.235175 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 11:42:30.235406 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260806 11:42:30.236441 14505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86: No bootstrap required, opened a new log
I20260806 11:42:30.240396 14505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daa90832d714aecbdd8d6eede8b0f86" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33099 } }
I20260806 11:42:30.240660 14505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:30.240728 14505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8daa90832d714aecbdd8d6eede8b0f86, State: Initialized, Role: FOLLOWER
I20260806 11:42:30.241266 14505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daa90832d714aecbdd8d6eede8b0f86" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33099 } }
I20260806 11:42:30.241444 14505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:42:30.241501 14505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:42:30.241573 14505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:30.242547 14505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daa90832d714aecbdd8d6eede8b0f86" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33099 } }
I20260806 11:42:30.242848 14505 leader_election.cc:302] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8daa90832d714aecbdd8d6eede8b0f86; no voters:
I20260806 11:42:30.243101 14505 leader_election.cc:288] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:42:30.243221 14512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:30.243475 14512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [term 1 LEADER]: Becoming Leader. State: Replica: 8daa90832d714aecbdd8d6eede8b0f86, State: Running, Role: LEADER
I20260806 11:42:30.243832 14512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daa90832d714aecbdd8d6eede8b0f86" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33099 } }
I20260806 11:42:30.244290 14505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:42:30.245517 14514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8daa90832d714aecbdd8d6eede8b0f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daa90832d714aecbdd8d6eede8b0f86" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33099 } } }
I20260806 11:42:30.245661 14514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:30.246068 14515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8daa90832d714aecbdd8d6eede8b0f86. Latest consensus state: current_term: 1 leader_uuid: "8daa90832d714aecbdd8d6eede8b0f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daa90832d714aecbdd8d6eede8b0f86" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33099 } } }
I20260806 11:42:30.246181 14515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:30.246068 14523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:42:30.249022 14523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:42:30.254240 14523 catalog_manager.cc:1347] Generated new cluster ID: 3560a2f81d38417f96cb570b2c3dc64a
I20260806 11:42:30.254305 14523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:42:30.275123 14523 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:42:30.275660 14523 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:42:30.284605 14523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8daa90832d714aecbdd8d6eede8b0f86: Generated new TSK 0
I20260806 11:42:30.285329 14523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:42:30.289517 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33099
--builtin_ntp_servers=127.0.113.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 11:42:30.403323 14538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:30.403553 14538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:30.403589 14538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:30.407661 14538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:30.407711 14538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:30.407794 14538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:42:30.412535 14538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:30.413687 14538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:30.415126 14538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:30.424646 14544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:30.424646 14549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:30.424667 14547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:30.425037 14538 server_base.cc:1034] running on GCE node
I20260806 11:42:30.425434 14538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:30.426081 14538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:30.427297 14538 hybrid_clock.cc:648] HybridClock initialized: now 1786016550427273 us; error 37 us; skew 500 ppm
I20260806 11:42:30.429256 14538 webserver.cc:460] Webserver started at http://127.0.113.1:42699/ using document root <none> and password file <none>
I20260806 11:42:30.429848 14538 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:30.429914 14538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:30.430135 14538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:30.432200 14538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "ffa9930f7e77402387d21925e91fb6df"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "80c8bf17e04f7642d525a098862f3533"
server_key_iv: "6015ccdcb1e0c56052e93c5ad5fe9338"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.432862 14538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "ffa9930f7e77402387d21925e91fb6df"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "80c8bf17e04f7642d525a098862f3533"
server_key_iv: "6015ccdcb1e0c56052e93c5ad5fe9338"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.436745 14538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 11:42:30.439209 14556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.440442 14538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:30.440558 14538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "ffa9930f7e77402387d21925e91fb6df"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "80c8bf17e04f7642d525a098862f3533"
server_key_iv: "6015ccdcb1e0c56052e93c5ad5fe9338"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.440671 14538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:30.445494 14538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:30.446180 14538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:30.446364 14538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:30.446954 14538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:30.447811 14538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:30.447860 14538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.447899 14538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:30.447914 14538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.456789 14538 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:38891
I20260806 11:42:30.456820 14718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:38891 every 8 connection(s)
I20260806 11:42:30.457646 14538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:42:30.463670 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:30.459454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:34705 (local address 127.0.113.62:33099)
0806 11:42:30.459591 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:30.459594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:30.460609 (+ 1015us) server_negotiation.cc:407] Connection header received
0806 11:42:30.461958 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:30.461962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:30.462010 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:30.462120 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:30.463189 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:30.463194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:30.463196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:42:30.463209 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:30.463236 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:42:30.463245 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:30.463337 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:30.463494 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 11:42:30.463538 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:42:30.464488 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:30.459782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33099 (local address 127.0.113.1:34705)
0806 11:42:30.460455 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:30.460494 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:30.461727 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:30.462274 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:30.462281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:30.462367 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:30.462928 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:42:30.462930 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:30.462978 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:42:30.463040 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:30.463332 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:30.463341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:30.463348 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 11:42:30.463535 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 11:42:30.463755 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260806 11:42:30.465670 14719 heartbeater.cc:347] Connected to a master server at 127.0.113.62:33099
I20260806 11:42:30.465940 14719 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:30.466339 14719 heartbeater.cc:511] Master 127.0.113.62:33099 requested a full tablet report, sending...
I20260806 11:42:30.466929 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 14538
I20260806 11:42:30.467092 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:42:30.467376 452 external_mini_cluster.cc:1428] Setting key aae2953dca655c68ff0f8ab2ac051f19
I20260806 11:42:30.468264 14448 ts_manager.cc:194] Registered new tserver with Master: ffa9930f7e77402387d21925e91fb6df (127.0.113.1:38891)
I20260806 11:42:30.469568 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33099
--builtin_ntp_servers=127.0.113.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 11:42:30.584288 14726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:30.584606 14726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:30.584647 14726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:30.588753 14726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:30.588832 14726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:30.588891 14726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:42:30.593803 14726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:30.595007 14726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:30.596506 14726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:30.606487 14735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:30.606532 14732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:30.606590 14726 server_base.cc:1034] running on GCE node
W20260806 11:42:30.606669 14731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:30.607270 14726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:30.608000 14726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:30.609218 14726 hybrid_clock.cc:648] HybridClock initialized: now 1786016550609190 us; error 41 us; skew 500 ppm
I20260806 11:42:30.611310 14726 webserver.cc:460] Webserver started at http://127.0.113.2:44749/ using document root <none> and password file <none>
I20260806 11:42:30.611939 14726 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:30.612042 14726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:30.612355 14726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:30.614692 14726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "d78976d3ab86480489b27f88ead21a04"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "50b0d4450d0972db971b47dfe239fbdd"
server_key_iv: "afd70696ae072aad0c67b0c72e353ae7"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.615330 14726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "d78976d3ab86480489b27f88ead21a04"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "50b0d4450d0972db971b47dfe239fbdd"
server_key_iv: "afd70696ae072aad0c67b0c72e353ae7"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.619257 14726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 11:42:30.622027 14744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.623315 14726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 11:42:30.623472 14726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "d78976d3ab86480489b27f88ead21a04"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "50b0d4450d0972db971b47dfe239fbdd"
server_key_iv: "afd70696ae072aad0c67b0c72e353ae7"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.623602 14726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:30.628872 14726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:30.629547 14726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:30.629758 14726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:30.630378 14726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:30.631394 14726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:30.631474 14726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.631553 14726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:30.631603 14726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.641419 14726 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:40781
I20260806 11:42:30.641500 14905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:40781 every 8 connection(s)
I20260806 11:42:30.642477 14726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:42:30.647001 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 14726
I20260806 11:42:30.647151 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:42:30.647477 452 external_mini_cluster.cc:1428] Setting key 7a9afe6f272358f1bd316df5c813d1f7
I20260806 11:42:30.649952 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33099
--builtin_ntp_servers=127.0.113.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 11:42:30.650146 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:30.644806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:49661 (local address 127.0.113.62:33099)
0806 11:42:30.644989 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:30.644992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:30.646383 (+ 1391us) server_negotiation.cc:407] Connection header received
0806 11:42:30.648102 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:30.648108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:30.648156 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:30.648252 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:30.649597 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:30.649600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:30.649602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:42:30.649613 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:30.649633 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:42:30.649641 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:30.649717 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:30.649934 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 11:42:30.649974 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 11:42:30.651409 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:30.645423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33099 (local address 127.0.113.2:49661)
0806 11:42:30.646161 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:30.646204 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:30.647710 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:30.648475 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:30.648482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:30.648571 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:30.649251 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:42:30.649254 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:30.649317 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:42:30.649419 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:30.649742 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:30.649755 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:30.649766 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 11:42:30.649980 (+ 214us) client_negotiation.cc:240] Negotiation successful
0806 11:42:30.650262 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":139,"threads_started":1}
I20260806 11:42:30.652712 14906 heartbeater.cc:347] Connected to a master server at 127.0.113.62:33099
I20260806 11:42:30.652966 14906 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:30.653445 14906 heartbeater.cc:511] Master 127.0.113.62:33099 requested a full tablet report, sending...
I20260806 11:42:30.654680 14448 ts_manager.cc:194] Registered new tserver with Master: d78976d3ab86480489b27f88ead21a04 (127.0.113.2:40781)
W20260806 11:42:30.777709 14912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:30.778072 14912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:30.778131 14912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:30.783702 14912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:30.783987 14912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:30.784070 14912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:42:30.789798 14912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:30.790968 14912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:30.792459 14912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:30.802650 14921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:30.802788 14922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:30.802642 14925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:30.803458 14912 server_base.cc:1034] running on GCE node
I20260806 11:42:30.803910 14912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:30.804677 14912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:30.805902 14912 hybrid_clock.cc:648] HybridClock initialized: now 1786016550805867 us; error 49 us; skew 500 ppm
I20260806 11:42:30.807973 14912 webserver.cc:460] Webserver started at http://127.0.113.3:41335/ using document root <none> and password file <none>
I20260806 11:42:30.808653 14912 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:30.808755 14912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:30.809012 14912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:30.811195 14912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "6b834b58496a4d5298bce83d60b23f64"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "092c4140b5b0a3e33b6decc9eccb0ea2"
server_key_iv: "4db3d7e8f48b95368e4a089442ce1578"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.811853 14912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "6b834b58496a4d5298bce83d60b23f64"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "092c4140b5b0a3e33b6decc9eccb0ea2"
server_key_iv: "4db3d7e8f48b95368e4a089442ce1578"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.816180 14912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 11:42:30.818811 14933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.820164 14912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 11:42:30.820354 14912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "6b834b58496a4d5298bce83d60b23f64"
format_stamp: "Formatted at 2026-08-06 11:42:30 on dist-test-slave-45dx"
server_key: "092c4140b5b0a3e33b6decc9eccb0ea2"
server_key_iv: "4db3d7e8f48b95368e4a089442ce1578"
server_key_version: "encryptionkey@0"
I20260806 11:42:30.820488 14912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:30.825819 14912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:30.826463 14912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:30.826654 14912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:30.827335 14912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:30.828446 14912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:30.828518 14912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.828584 14912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:30.828634 14912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:30.839859 14912 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:38081
I20260806 11:42:30.839880 15090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:38081 every 8 connection(s)
I20260806 11:42:30.840909 14912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:42:30.845706 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 14912
I20260806 11:42:30.845850 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:42:30.846231 452 external_mini_cluster.cc:1428] Setting key 23066b6a9f9a89c91147c6e3c6e12488
I20260806 11:42:30.848624 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:30.843138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:57121 (local address 127.0.113.62:33099)
0806 11:42:30.843431 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:30.843439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:30.844670 (+ 1231us) server_negotiation.cc:407] Connection header received
0806 11:42:30.846430 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:30.846433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:30.846481 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:30.846593 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:30.847903 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:30.847908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:30.847913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:42:30.847931 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:30.847962 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:42:30.847975 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:30.848096 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:30.848385 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 11:42:30.848445 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260806 11:42:30.850318 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:30.843560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33099 (local address 127.0.113.3:57121)
0806 11:42:30.844475 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:30.844564 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:30.846159 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:30.846770 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:30.846776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:30.846902 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:30.847585 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:42:30.847587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:30.847644 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:42:30.847735 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:30.848152 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:30.848164 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:30.848171 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 11:42:30.848918 (+ 747us) client_negotiation.cc:240] Negotiation successful
0806 11:42:30.849188 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":148,"threads_started":1}
I20260806 11:42:30.851554 15091 heartbeater.cc:347] Connected to a master server at 127.0.113.62:33099
I20260806 11:42:30.851802 15091 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:30.852243 15091 heartbeater.cc:511] Master 127.0.113.62:33099 requested a full tablet report, sending...
I20260806 11:42:30.853297 14448 ts_manager.cc:194] Registered new tserver with Master: 6b834b58496a4d5298bce83d60b23f64 (127.0.113.3:38081)
I20260806 11:42:30.861033 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 11:42:30.864149 14510 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:42:30.863104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.0.113.62:33099)
0806 11:42:30.863235 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:30.863239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:30.863443 (+ 204us) server_negotiation.cc:407] Connection header received
0806 11:42:30.863550 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:30.863552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:30.863585 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:30.863654 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:30.864016 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57694: BlockingRecv error: recv got EOF from 127.0.0.1:57694 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 11:42:30.864485 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 14538
I20260806 11:42:30.875949 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 14726
I20260806 11:42:30.889873 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 14912
I20260806 11:42:30.904748 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 14404
2026-08-06T11:42:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (874 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:42:30 dist-test-slave-45dx krb5kdc[15114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:42:30 dist-test-slave-45dx krb5kdc[15114](info): set up 2 sockets
Aug 06 11:42:30 dist-test-slave-45dx krb5kdc[15114](info): commencing operation
krb5kdc: starting...
W20260806 11:42:32.186113 452 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:42:32 dist-test-slave-45dx krb5kdc[15114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:42:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:42:32.320048 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:42883
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:42883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:32.433156 15130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:32.433514 15130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:32.433601 15130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:32.438017 15130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:42:32.438135 15130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:32.438179 15130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:32.438243 15130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:42:32.438284 15130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:42:32.443655 15130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:42883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:42883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:32.444871 15130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:32.445907 15130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:32.454782 15137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:32.454831 15140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:32.454774 15138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:32.455041 15130 server_base.cc:1034] running on GCE node
I20260806 11:42:32.455686 15130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:32.456827 15130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:32.457998 15130 hybrid_clock.cc:648] HybridClock initialized: now 1786016552457985 us; error 26 us; skew 500 ppm
Aug 06 11:42:32 dist-test-slave-45dx krb5kdc[15114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:32.460919 15130 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:42:32.461807 15130 webserver.cc:460] Webserver started at http://127.0.113.62:33047/ using document root <none> and password file <none>
I20260806 11:42:32.462399 15130 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:32.462494 15130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:32.462733 15130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:32.464839 15130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "d00403fec7394c17a90a174c6b143040"
format_stamp: "Formatted at 2026-08-06 11:42:32 on dist-test-slave-45dx"
server_key: "ba41d961898adbed3841ab18b534c3a2"
server_key_iv: "5f7024000788b5d79526c4a61371901d"
server_key_version: "encryptionkey@0"
I20260806 11:42:32.465416 15130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "d00403fec7394c17a90a174c6b143040"
format_stamp: "Formatted at 2026-08-06 11:42:32 on dist-test-slave-45dx"
server_key: "ba41d961898adbed3841ab18b534c3a2"
server_key_iv: "5f7024000788b5d79526c4a61371901d"
server_key_version: "encryptionkey@0"
I20260806 11:42:32.469097 15130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:42:32.471388 15147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:32.472710 15130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:42:32.472854 15130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "d00403fec7394c17a90a174c6b143040"
format_stamp: "Formatted at 2026-08-06 11:42:32 on dist-test-slave-45dx"
server_key: "ba41d961898adbed3841ab18b534c3a2"
server_key_iv: "5f7024000788b5d79526c4a61371901d"
server_key_version: "encryptionkey@0"
I20260806 11:42:32.473004 15130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:32.490582 15130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:32.493518 15130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:32.493732 15130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:32.500891 15130 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:42883
I20260806 11:42:32.501174 15229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:42883 every 8 connection(s)
I20260806 11:42:32.501904 15130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:42:32.504698 15230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:32.507014 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 15130
I20260806 11:42:32.507153 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:42:32.507432 452 external_mini_cluster.cc:1428] Setting key 906bf34ba3a0f1c7126b81329f1ee988
I20260806 11:42:32.510461 15230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040: Bootstrap starting.
I20260806 11:42:32.513044 15230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:32.513881 15230 log.cc:824] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:32.515895 15230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040: No bootstrap required, opened a new log
Aug 06 11:42:32 dist-test-slave-45dx krb5kdc[15114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:42:32.520459 15230 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00403fec7394c17a90a174c6b143040" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42883 } }
I20260806 11:42:32.520759 15230 raft_consensus.cc:374] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:32.520828 15230 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00403fec7394c17a90a174c6b143040, State: Initialized, Role: FOLLOWER
I20260806 11:42:32.521260 15230 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00403fec7394c17a90a174c6b143040" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42883 } }
I20260806 11:42:32.521405 15230 raft_consensus.cc:388] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:42:32.521493 15230 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:42:32.521615 15230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:32.521916 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:32.508774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.0.113.62:42883)
0806 11:42:32.509215 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:32.509235 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:32.509285 (+ 50us) server_negotiation.cc:407] Connection header received
0806 11:42:32.509921 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:32.509937 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:32.510241 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:32.510521 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:32.511438 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:32.512481 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:32.513590 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:32.514081 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:32.514193 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:32.517072 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:32.517077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:32.517091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:32.517118 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:32.518836 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:32.519325 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:32.519330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:32.519334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:32.519456 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:32.519746 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:32.519749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:32.519750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:32.520122 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:32.520289 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:32.520606 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 11:42:32.520910 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":133,"threads_started":1}
I20260806 11:42:32.522632 15230 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00403fec7394c17a90a174c6b143040" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42883 } }
I20260806 11:42:32.523006 15230 leader_election.cc:302] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d00403fec7394c17a90a174c6b143040; no voters:
I20260806 11:42:32.523355 15230 leader_election.cc:288] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:42:32.523455 15236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:32.523728 15236 raft_consensus.cc:686] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [term 1 LEADER]: Becoming Leader. State: Replica: d00403fec7394c17a90a174c6b143040, State: Running, Role: LEADER
I20260806 11:42:32.524035 15236 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00403fec7394c17a90a174c6b143040" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42883 } }
I20260806 11:42:32.524660 15230 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:42:32.525416 15238 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00403fec7394c17a90a174c6b143040. Latest consensus state: current_term: 1 leader_uuid: "d00403fec7394c17a90a174c6b143040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00403fec7394c17a90a174c6b143040" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42883 } } }
I20260806 11:42:32.525545 15238 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:32.525837 15237 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d00403fec7394c17a90a174c6b143040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00403fec7394c17a90a174c6b143040" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 42883 } } }
I20260806 11:42:32.525944 15237 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040 [sys.catalog]: This master's current role is: LEADER
I20260806 11:42:32.525967 15249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:42:32.529150 15249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:42:32.534655 15249 catalog_manager.cc:1347] Generated new cluster ID: 4277afdb6967421eb9f9f97baf6465b6
I20260806 11:42:32.534757 15249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:42:32.572808 15249 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:42:32.573824 15249 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:42:32.577515 15249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040: Generated new TSK 0
I20260806 11:42:32.578342 15249 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:42:32.619743 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42883
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:32.732170 15265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:32.732530 15265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:32.732581 15265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:32.736580 15265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:32.736644 15265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:32.736701 15265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:42:32.741916 15265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:32.742961 15265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:32.743965 15265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:32.754040 15277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:32.754041 15276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:32.754170 15265 server_base.cc:1034] running on GCE node
W20260806 11:42:32.754065 15279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:32.754773 15265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:32.755460 15265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:32.756647 15265 hybrid_clock.cc:648] HybridClock initialized: now 1786016552756616 us; error 52 us; skew 500 ppm
Aug 06 11:42:32 dist-test-slave-45dx krb5kdc[15114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:32.759444 15265 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:42:32.760349 15265 webserver.cc:460] Webserver started at http://127.0.113.1:34661/ using document root <none> and password file <none>
I20260806 11:42:32.760939 15265 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:32.761035 15265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:32.761274 15265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:32.763312 15265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "e9f2fa8358f6457aba4fda49885e3943"
format_stamp: "Formatted at 2026-08-06 11:42:32 on dist-test-slave-45dx"
server_key: "8c66770ef880ca3984fa8acf89e8c18d"
server_key_iv: "cac544cca1f42db8c5d44979872061f0"
server_key_version: "encryptionkey@0"
I20260806 11:42:32.763907 15265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "e9f2fa8358f6457aba4fda49885e3943"
format_stamp: "Formatted at 2026-08-06 11:42:32 on dist-test-slave-45dx"
server_key: "8c66770ef880ca3984fa8acf89e8c18d"
server_key_iv: "cac544cca1f42db8c5d44979872061f0"
server_key_version: "encryptionkey@0"
I20260806 11:42:32.767611 15265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:42:32.769964 15292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:32.771217 15265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:42:32.771349 15265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "e9f2fa8358f6457aba4fda49885e3943"
format_stamp: "Formatted at 2026-08-06 11:42:32 on dist-test-slave-45dx"
server_key: "8c66770ef880ca3984fa8acf89e8c18d"
server_key_iv: "cac544cca1f42db8c5d44979872061f0"
server_key_version: "encryptionkey@0"
I20260806 11:42:32.771468 15265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:32.792649 15265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:32.795626 15265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:32.795840 15265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:32.796537 15265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:32.797348 15265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:32.797421 15265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:32.797495 15265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:32.797538 15265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:32.806221 15265 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:38665
I20260806 11:42:32.806248 15478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:38665 every 8 connection(s)
I20260806 11:42:32.807096 15265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
Aug 06 11:42:32 dist-test-slave-45dx krb5kdc[15114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:42:32.817025 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 15265
I20260806 11:42:32.817132 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:42:32.817449 452 external_mini_cluster.cc:1428] Setting key a64c5d24d2aae013aed0a0e5a3c2eba7
I20260806 11:42:32.821786 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:32.809064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:33857 (local address 127.0.113.62:42883)
0806 11:42:32.809230 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:32.809235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:32.810047 (+ 812us) server_negotiation.cc:407] Connection header received
0806 11:42:32.811310 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:32.811313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:32.811362 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:32.811443 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:32.812711 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:32.813253 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:32.814488 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:32.814715 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:32.814798 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:32.817539 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:32.817543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:32.817546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:32.817603 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:32.819459 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:32.819961 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:32.819967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:32.819971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:32.820065 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:32.820444 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:32.820450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:32.820455 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:32.820717 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:32.820826 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:32.821308 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 11:42:32.821510 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 11:42:32.823110 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:32.809439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:42883 (local address 127.0.113.1:33857)
0806 11:42:32.809895 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:32.809940 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:32.811090 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:32.811609 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:32.811615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:32.811988 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:32.812532 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:32.812561 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:32.813459 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:32.813463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:32.814359 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:32.814369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:32.814867 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:32.814870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:32.815011 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:32.815571 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:32.815587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:32.817305 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:32.819575 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:32.819580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:32.819590 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:32.819858 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:32.820162 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:32.820164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:32.820165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:32.820298 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:32.820829 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:32.820836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:32.821060 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 11:42:32.821211 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 11:42:32.821596 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260806 11:42:32.824469 15479 heartbeater.cc:347] Connected to a master server at 127.0.113.62:42883
I20260806 11:42:32.824765 15479 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:32.825444 15479 heartbeater.cc:511] Master 127.0.113.62:42883 requested a full tablet report, sending...
I20260806 11:42:32.827111 15169 ts_manager.cc:194] Registered new tserver with Master: e9f2fa8358f6457aba4fda49885e3943 (127.0.113.1:38665)
I20260806 11:42:32.828282 15169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:33857
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:42:32.847028 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42883
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:32.955510 15490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:32.955737 15490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:32.955776 15490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:32.959664 15490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:32.959717 15490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:32.959800 15490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:42:32.964882 15490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:32.965813 15490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:32.966773 15490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:42:32.976200 15500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:32.976208 15504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:32.976213 15501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:32.976508 15490 server_base.cc:1034] running on GCE node
I20260806 11:42:32.976954 15490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:32.977571 15490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:32.978739 15490 hybrid_clock.cc:648] HybridClock initialized: now 1786016552978720 us; error 25 us; skew 500 ppm
Aug 06 11:42:32 dist-test-slave-45dx krb5kdc[15114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:32.981590 15490 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:42:32.982365 15490 webserver.cc:460] Webserver started at http://127.0.113.2:45969/ using document root <none> and password file <none>
I20260806 11:42:32.982853 15490 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:32.982919 15490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:32.983074 15490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:32.984987 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "d85e5fbc53a04502a4975c4334358cc0"
format_stamp: "Formatted at 2026-08-06 11:42:32 on dist-test-slave-45dx"
server_key: "2fe6e909e50dcc4d32552320419cdbd0"
server_key_iv: "50a26f576881bead5cf21f56c9b3c19d"
server_key_version: "encryptionkey@0"
I20260806 11:42:32.985463 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "d85e5fbc53a04502a4975c4334358cc0"
format_stamp: "Formatted at 2026-08-06 11:42:32 on dist-test-slave-45dx"
server_key: "2fe6e909e50dcc4d32552320419cdbd0"
server_key_iv: "50a26f576881bead5cf21f56c9b3c19d"
server_key_version: "encryptionkey@0"
I20260806 11:42:32.988994 15490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 11:42:32.991214 15514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:32.992419 15490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 11:42:32.992542 15490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "d85e5fbc53a04502a4975c4334358cc0"
format_stamp: "Formatted at 2026-08-06 11:42:32 on dist-test-slave-45dx"
server_key: "2fe6e909e50dcc4d32552320419cdbd0"
server_key_iv: "50a26f576881bead5cf21f56c9b3c19d"
server_key_version: "encryptionkey@0"
I20260806 11:42:32.992645 15490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:33.006053 15490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:33.008790 15490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:33.009001 15490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:33.009549 15490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:33.010339 15490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:33.010412 15490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:33.010495 15490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:33.010543 15490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:33.019557 15490 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:45651
I20260806 11:42:33.019578 15678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:45651 every 8 connection(s)
I20260806 11:42:33.020525 15490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:42:33.022949 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 15490
I20260806 11:42:33.023092 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:42:33.023372 452 external_mini_cluster.cc:1428] Setting key 05ccc323cf27e667187f090a6bb6f1fa
Aug 06 11:42:33 dist-test-slave-45dx krb5kdc[15114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:42:33.035895 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.022446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:57039 (local address 127.0.113.62:42883)
0806 11:42:33.022590 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:33.022594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:33.023455 (+ 861us) server_negotiation.cc:407] Connection header received
0806 11:42:33.025002 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:33.025005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:33.025082 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:33.025149 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:33.026588 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.027337 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.028558 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.028942 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.029070 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.031907 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:33.031915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:33.031922 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:33.031960 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:33.033762 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:33.034275 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:33.034280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:33.034282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:33.034379 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:33.034779 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:33.034785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:33.034788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:33.035014 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:33.035129 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:33.035645 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 11:42:33.035766 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:42:33.036854 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.022818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:42883 (local address 127.0.113.2:57039)
0806 11:42:33.023304 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:33.023357 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:33.024795 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:33.025360 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:33.025369 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:33.025887 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:33.026448 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.026463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.027495 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.027498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.028437 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.028447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.029112 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.029115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.029252 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.029883 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:33.029901 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:33.031719 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:33.033886 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:33.033892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:33.033906 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:33.034167 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:33.034508 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:33.034510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:33.034511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:33.034659 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:33.035155 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:33.035163 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:33.035415 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 11:42:33.035573 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 11:42:33.035842 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
I20260806 11:42:33.038293 15679 heartbeater.cc:347] Connected to a master server at 127.0.113.62:42883
I20260806 11:42:33.038558 15679 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:33.039073 15679 heartbeater.cc:511] Master 127.0.113.62:42883 requested a full tablet report, sending...
I20260806 11:42:33.040216 15169 ts_manager.cc:194] Registered new tserver with Master: d85e5fbc53a04502a4975c4334358cc0 (127.0.113.2:45651)
I20260806 11:42:33.040854 15169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:57039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:42:33.055053 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42883
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:42:33.168234 15686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:42:33.168519 15686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:42:33.168560 15686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:42:33.172678 15686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:42:33.172746 15686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:42:33.172828 15686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:42:33.178426 15686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:42883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:42:33.179468 15686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:42:33.180548 15686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 11:42:33.190798 15686 server_base.cc:1034] running on GCE node
W20260806 11:42:33.190770 15691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:33.190758 15694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:42:33.190733 15692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:42:33.191470 15686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:42:33.192137 15686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:42:33.193325 15686 hybrid_clock.cc:648] HybridClock initialized: now 1786016553193320 us; error 39 us; skew 500 ppm
Aug 06 11:42:33 dist-test-slave-45dx krb5kdc[15114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:42:33.196393 15686 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:42:33.197218 15686 webserver.cc:460] Webserver started at http://127.0.113.3:38805/ using document root <none> and password file <none>
I20260806 11:42:33.197768 15686 fs_manager.cc:359] Metadata directory not provided
I20260806 11:42:33.197829 15686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:42:33.197991 15686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:42:33.199961 15686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "89c76f29b2f64f57a522e0ee1c7cbed3"
format_stamp: "Formatted at 2026-08-06 11:42:33 on dist-test-slave-45dx"
server_key: "f8225fd8257723af0a05909faaefdf7a"
server_key_iv: "7681f87aa881595661728a90bae37514"
server_key_version: "encryptionkey@0"
I20260806 11:42:33.200543 15686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "89c76f29b2f64f57a522e0ee1c7cbed3"
format_stamp: "Formatted at 2026-08-06 11:42:33 on dist-test-slave-45dx"
server_key: "f8225fd8257723af0a05909faaefdf7a"
server_key_iv: "7681f87aa881595661728a90bae37514"
server_key_version: "encryptionkey@0"
I20260806 11:42:33.204530 15686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 11:42:33.207140 15702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:33.208454 15686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:42:33.208597 15686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "89c76f29b2f64f57a522e0ee1c7cbed3"
format_stamp: "Formatted at 2026-08-06 11:42:33 on dist-test-slave-45dx"
server_key: "f8225fd8257723af0a05909faaefdf7a"
server_key_iv: "7681f87aa881595661728a90bae37514"
server_key_version: "encryptionkey@0"
I20260806 11:42:33.208712 15686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:42:33.221725 15686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:42:33.224651 15686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:42:33.224860 15686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:42:33.225445 15686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:42:33.226377 15686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:42:33.226467 15686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:33.226547 15686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:42:33.226599 15686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:42:33.236117 15686 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:45405
I20260806 11:42:33.236150 15868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:45405 every 8 connection(s)
I20260806 11:42:33.237035 15686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:42:33.242328 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 15686
I20260806 11:42:33.242470 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:42:33.242774 452 external_mini_cluster.cc:1428] Setting key d20875f20f5d0985202fbab580c5f550
Aug 06 11:42:33 dist-test-slave-45dx krb5kdc[15114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:42:33.250914 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.238873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:42387 (local address 127.0.113.62:42883)
0806 11:42:33.239034 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:33.239037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:33.239860 (+ 823us) server_negotiation.cc:407] Connection header received
0806 11:42:33.241187 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:33.241191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:33.241249 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:33.241328 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:33.242591 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.243202 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.244834 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.245122 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.245201 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.247702 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:33.247705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:33.247707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:33.247731 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:33.249137 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:33.249616 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:33.249619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:33.249621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:33.249693 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:33.249994 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:33.250004 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:33.250005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:33.250167 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:33.250241 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:33.250700 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 11:42:33.250794 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 11:42:33.251735 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.239241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:42883 (local address 127.0.113.3:42387)
0806 11:42:33.239702 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:33.239745 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:33.240972 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:33.241492 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:33.241505 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:33.241881 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:42:33.242354 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.242367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.243340 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.243343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.244718 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.244729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.245265 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.245268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.245413 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.246091 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:42:33.246108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:42:33.247544 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:42:33.249240 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:33.249245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:33.249256 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:33.249530 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:33.249774 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:42:33.249776 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:42:33.249778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:42:33.249916 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:42:33.250254 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:42:33.250258 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:42:33.250495 (+ 237us) client_negotiation.cc:769] Sending connection context
0806 11:42:33.250653 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 11:42:33.250905 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":128,"threads_started":1}
I20260806 11:42:33.252959 15871 heartbeater.cc:347] Connected to a master server at 127.0.113.62:42883
I20260806 11:42:33.253170 15871 heartbeater.cc:464] Registering TS with master...
I20260806 11:42:33.253611 15871 heartbeater.cc:511] Master 127.0.113.62:42883 requested a full tablet report, sending...
I20260806 11:42:33.254601 15169 ts_manager.cc:194] Registered new tserver with Master: 89c76f29b2f64f57a522e0ee1c7cbed3 (127.0.113.3:45405)
I20260806 11:42:33.255122 15169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:42387
I20260806 11:42:33.256887 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:42:33.265532 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.258898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.0.113.62:42883)
0806 11:42:33.259015 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:33.259018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:33.259198 (+ 180us) server_negotiation.cc:407] Connection header received
0806 11:42:33.259289 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:33.259291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:33.259326 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:33.259386 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:42:33.260020 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.260563 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.261422 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.261627 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.261706 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.262947 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:42:33.262950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:42:33.262952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:42:33.262986 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:42:33.264151 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:33.264594 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:33.264596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:33.264598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:33.264670 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:42:33.264883 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:42:33.264885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:42:33.264886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:42:33.265068 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:42:33.265166 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:33.265325 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 11:42:33.265421 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:42:33.268270 15169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:42:33.271243 15169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:42:33.286536 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.279457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:45405 (local address 127.0.0.1:53036)
0806 11:42:33.279810 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:33.279856 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:33.279972 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:33.282206 (+ 2234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:33.282209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:33.282248 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:33.282347 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.282353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.283983 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.283988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.284903 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.284913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.285953 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.285957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.286076 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.286122 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 11:42:33.286237 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 11:42:33.286374 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"spinlock_wait_cycles":3328,"thread_start_us":120,"threads_started":1}
I20260806 11:42:33.286872 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.279589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.0.113.3:45405)
0806 11:42:33.281699 (+ 2110us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:33.281715 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:33.281739 (+ 24us) server_negotiation.cc:407] Connection header received
0806 11:42:33.281830 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:33.281835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:33.282031 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:33.282510 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:33.282668 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.283802 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.285036 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.285819 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.285951 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.286106 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:33.286622 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 11:42:33.286724 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":135,"threads_started":1}
I20260806 11:42:33.287267 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.280538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.0.113.1:38665)
0806 11:42:33.281309 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:33.281334 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:33.281362 (+ 28us) server_negotiation.cc:407] Connection header received
0806 11:42:33.281444 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:33.281450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:33.281689 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:33.281879 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:33.282193 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.283346 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.285284 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.286078 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.286181 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.286311 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:33.286433 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 11:42:33.286525 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":153,"threads_started":1}
I20260806 11:42:33.287406 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.281047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.0.113.2:45651)
0806 11:42:33.281529 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:33.281540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:33.282348 (+ 808us) server_negotiation.cc:407] Connection header received
0806 11:42:33.282427 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:33.282431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:33.282601 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:33.282743 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:33.283144 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.284244 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.285583 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.286286 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.286381 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.286514 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:33.287145 (+ 631us) server_negotiation.cc:299] Negotiation successful
0806 11:42:33.287256 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":110,"threads_started":1}
I20260806 11:42:33.287647 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.280577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:45651 (local address 127.0.0.1:39310)
0806 11:42:33.282141 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:33.282155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:33.282257 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:33.282906 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:33.282909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:33.282926 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:33.283012 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.283017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.284425 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.284428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.285467 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.285474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.286585 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.286589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.286693 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.286729 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 11:42:33.286793 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 11:42:33.286853 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1491,"mutex_wait_us":4,"spinlock_wait_cycles":11008,"thread_start_us":117,"threads_started":1}
I20260806 11:42:33.287674 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.280203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:38665 (local address 127.0.0.1:34476)
0806 11:42:33.280522 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:33.280540 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:33.280684 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:33.281870 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:33.281877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:33.281900 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:33.282051 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.282060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.283754 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.283757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.285142 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.285153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.286188 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.286191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.286287 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.286310 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 11:42:33.286394 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 11:42:33.286469 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":21,"thread_start_us":125,"threads_started":1}
I20260806 11:42:33.290024 15372 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdcce97cf26424498a14bd803977f87 (DEFAULT_TABLE table=test-table [id=a386d3513678480e917538eaf4717633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:33.290084 15766 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdcce97cf26424498a14bd803977f87 (DEFAULT_TABLE table=test-table [id=a386d3513678480e917538eaf4717633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:33.290334 15586 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdcce97cf26424498a14bd803977f87 (DEFAULT_TABLE table=test-table [id=a386d3513678480e917538eaf4717633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:42:33.291108 15372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdcce97cf26424498a14bd803977f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:33.291178 15766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdcce97cf26424498a14bd803977f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:33.291195 15586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdcce97cf26424498a14bd803977f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:42:33.297521 15894 tablet_bootstrap.cc:492] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943: Bootstrap starting.
I20260806 11:42:33.298853 15898 tablet_bootstrap.cc:492] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: Bootstrap starting.
I20260806 11:42:33.299731 15894 tablet_bootstrap.cc:654] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:33.301128 15894 log.cc:824] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:33.301170 15898 tablet_bootstrap.cc:654] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:33.301828 15898 log.cc:824] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:33.302534 15895 tablet_bootstrap.cc:492] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: Bootstrap starting.
I20260806 11:42:33.303051 15894 tablet_bootstrap.cc:492] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943: No bootstrap required, opened a new log
I20260806 11:42:33.303251 15894 ts_tablet_manager.cc:1397] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 11:42:33.303771 15898 tablet_bootstrap.cc:492] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: No bootstrap required, opened a new log
I20260806 11:42:33.303949 15898 ts_tablet_manager.cc:1397] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:42:33.304471 15895 tablet_bootstrap.cc:654] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: Neither blocks nor log segments found. Creating new log.
I20260806 11:42:33.305423 15895 log.cc:824] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: Log is configured to *not* fsync() on all Append() calls
I20260806 11:42:33.307331 15895 tablet_bootstrap.cc:492] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: No bootstrap required, opened a new log
I20260806 11:42:33.307191 15894 raft_consensus.cc:348] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } }
I20260806 11:42:33.307524 15894 raft_consensus.cc:374] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:33.307540 15895 ts_tablet_manager.cc:1397] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 11:42:33.307593 15894 raft_consensus.cc:729] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f2fa8358f6457aba4fda49885e3943, State: Initialized, Role: FOLLOWER
I20260806 11:42:33.307652 15898 raft_consensus.cc:348] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } }
I20260806 11:42:33.307940 15898 raft_consensus.cc:374] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:33.308034 15898 raft_consensus.cc:729] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85e5fbc53a04502a4975c4334358cc0, State: Initialized, Role: FOLLOWER
I20260806 11:42:33.308101 15894 consensus_queue.cc:260] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } }
I20260806 11:42:33.308501 15898 consensus_queue.cc:260] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } }
I20260806 11:42:33.308938 15479 heartbeater.cc:503] Master 127.0.113.62:42883 was elected leader, sending a full tablet report...
I20260806 11:42:33.309321 15894 ts_tablet_manager.cc:1428] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 11:42:33.309406 15898 ts_tablet_manager.cc:1428] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 11:42:33.309904 15679 heartbeater.cc:503] Master 127.0.113.62:42883 was elected leader, sending a full tablet report...
I20260806 11:42:33.311345 15895 raft_consensus.cc:348] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } }
I20260806 11:42:33.311615 15895 raft_consensus.cc:374] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:42:33.311708 15895 raft_consensus.cc:729] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c76f29b2f64f57a522e0ee1c7cbed3, State: Initialized, Role: FOLLOWER
I20260806 11:42:33.312173 15895 consensus_queue.cc:260] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } }
I20260806 11:42:33.312971 15871 heartbeater.cc:503] Master 127.0.113.62:42883 was elected leader, sending a full tablet report...
I20260806 11:42:33.313314 15895 ts_tablet_manager.cc:1428] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 11:42:33.470470 15905 raft_consensus.cc:482] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:42:33.470723 15905 raft_consensus.cc:504] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } }
I20260806 11:42:33.472023 15905 leader_election.cc:288] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89c76f29b2f64f57a522e0ee1c7cbed3 (127.0.113.3:45405), e9f2fa8358f6457aba4fda49885e3943 (127.0.113.1:38665)
I20260806 11:42:33.476408 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.472529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:39505 (local address 127.0.113.3:45405)
0806 11:42:33.472651 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:33.472656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:33.472703 (+ 47us) server_negotiation.cc:407] Connection header received
0806 11:42:33.472760 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:33.472763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:33.472822 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:33.472903 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:33.473275 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.473926 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.475100 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.475738 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.475822 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.475876 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:33.476116 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 11:42:33.476242 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 11:42:33.476347 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.472350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:45405 (local address 127.0.113.2:39505)
0806 11:42:33.472529 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:33.472548 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:33.472634 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:33.472983 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:33.472986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:33.473011 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:33.473142 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.473150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.474040 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.474043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.474980 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.474991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.475853 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.475857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.475968 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.475994 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:42:33.476074 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 11:42:33.476144 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":3}
I20260806 11:42:33.476774 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.472772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:58391 (local address 127.0.113.1:38665)
0806 11:42:33.472935 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:33.472942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:33.473143 (+ 201us) server_negotiation.cc:407] Connection header received
0806 11:42:33.473239 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:33.473245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:33.473306 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:33.473365 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:42:33.473658 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.474219 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.475520 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.476152 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.476247 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.476338 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:33.476574 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 11:42:33.476666 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 11:42:33.477164 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.472640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:38665 (local address 127.0.113.2:58391)
0806 11:42:33.473037 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:33.473056 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 11:42:33.473142 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:42:33.473402 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:42:33.473404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:42:33.473421 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:42:33.473544 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.473550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.474359 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.474362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.475389 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:42:33.475400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.476270 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.476273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:42:33.476405 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.476439 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 11:42:33.476934 (+ 495us) client_negotiation.cc:240] Negotiation successful
0806 11:42:33.477020 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":171,"threads_started":1}
I20260806 11:42:33.477487 15794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdcce97cf26424498a14bd803977f87" candidate_uuid: "d85e5fbc53a04502a4975c4334358cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" is_pre_election: true
I20260806 11:42:33.477826 15794 raft_consensus.cc:2454] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85e5fbc53a04502a4975c4334358cc0 in term 0.
I20260806 11:42:33.478304 15520 leader_election.cc:302] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89c76f29b2f64f57a522e0ee1c7cbed3, d85e5fbc53a04502a4975c4334358cc0; no voters:
I20260806 11:42:33.478614 15905 raft_consensus.cc:2781] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:42:33.478634 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdcce97cf26424498a14bd803977f87" candidate_uuid: "d85e5fbc53a04502a4975c4334358cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f2fa8358f6457aba4fda49885e3943" is_pre_election: true
I20260806 11:42:33.478727 15905 raft_consensus.cc:482] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:42:33.478793 15905 raft_consensus.cc:3037] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:33.478914 15402 raft_consensus.cc:2454] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85e5fbc53a04502a4975c4334358cc0 in term 0.
I20260806 11:42:33.480067 15905 raft_consensus.cc:504] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } }
I20260806 11:42:33.480522 15905 leader_election.cc:288] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 89c76f29b2f64f57a522e0ee1c7cbed3 (127.0.113.3:45405), e9f2fa8358f6457aba4fda49885e3943 (127.0.113.1:38665)
I20260806 11:42:33.480896 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdcce97cf26424498a14bd803977f87" candidate_uuid: "d85e5fbc53a04502a4975c4334358cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f2fa8358f6457aba4fda49885e3943"
I20260806 11:42:33.481010 15402 raft_consensus.cc:3037] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:33.482110 15402 raft_consensus.cc:2454] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85e5fbc53a04502a4975c4334358cc0 in term 1.
I20260806 11:42:33.482414 15520 leader_election.cc:302] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d85e5fbc53a04502a4975c4334358cc0, e9f2fa8358f6457aba4fda49885e3943; no voters:
I20260806 11:42:33.482401 15794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdcce97cf26424498a14bd803977f87" candidate_uuid: "d85e5fbc53a04502a4975c4334358cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3"
I20260806 11:42:33.482540 15794 raft_consensus.cc:3037] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:42:33.482594 15905 raft_consensus.cc:2781] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:42:33.482815 15905 raft_consensus.cc:686] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 1 LEADER]: Becoming Leader. State: Replica: d85e5fbc53a04502a4975c4334358cc0, State: Running, Role: LEADER
I20260806 11:42:33.483180 15905 consensus_queue.cc:237] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } }
I20260806 11:42:33.483851 15794 raft_consensus.cc:2454] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85e5fbc53a04502a4975c4334358cc0 in term 1.
I20260806 11:42:33.486812 15167 catalog_manager.cc:5495] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d85e5fbc53a04502a4975c4334358cc0 (127.0.113.2). New cstate: current_term: 1 leader_uuid: "d85e5fbc53a04502a4975c4334358cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d85e5fbc53a04502a4975c4334358cc0" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 45651 } health_report { overall_health: HEALTHY } } }
W20260806 11:42:33.487947 15873 tablet.cc:2369] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:42:33.521700 15681 tablet.cc:2369] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:42:33.526618 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.522982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.0.113.2:45651)
0806 11:42:33.523135 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 11:42:33.523140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:42:33.523156 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:42:33.523251 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:42:33.523253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:42:33.523310 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:42:33.523429 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:42:33.523686 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.524196 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.525104 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:42:33.525326 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:42:33.525464 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:42:33.525782 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:42:33.526193 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:42:33.526327 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 11:42:33.526387 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 11:42:33.526448 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 11:42:33.532454 15402 raft_consensus.cc:1261] T 8fdcce97cf26424498a14bd803977f87 P e9f2fa8358f6457aba4fda49885e3943 [term 1 FOLLOWER]: Refusing update from remote peer d85e5fbc53a04502a4975c4334358cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:33.532459 15794 raft_consensus.cc:1261] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [term 1 FOLLOWER]: Refusing update from remote peer d85e5fbc53a04502a4975c4334358cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:42:33.533267 15911 consensus_queue.cc:1035] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c76f29b2f64f57a522e0ee1c7cbed3" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:33.533515 15905 consensus_queue.cc:1035] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f2fa8358f6457aba4fda49885e3943" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:42:33.549652 15167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:33.549854 15167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:42:33.552448 15167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d00403fec7394c17a90a174c6b143040: Sending DeleteTablet for 3 replicas of tablet 8fdcce97cf26424498a14bd803977f87
I20260806 11:42:33.553314 15766 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdcce97cf26424498a14bd803977f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:33 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:53036
I20260806 11:42:33.553381 15586 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdcce97cf26424498a14bd803977f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:33 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:39310
I20260806 11:42:33.553896 15929 tablet_replica.cc:330] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: stopping tablet replica
I20260806 11:42:33.554031 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 15265
I20260806 11:42:33.554097 15372 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdcce97cf26424498a14bd803977f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:42:33 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:34476
I20260806 11:42:33.561937 15929 raft_consensus.cc:2229] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:42:33.562340 15929 raft_consensus.cc:2258] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:33.564468 15928 tablet_replica.cc:330] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: stopping tablet replica
I20260806 11:42:33.564905 15928 raft_consensus.cc:2229] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:42:33.565141 15929 ts_tablet_manager.cc:1905] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:33.565387 15928 raft_consensus.cc:2258] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:42:33.567028 15928 ts_tablet_manager.cc:1905] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:42:33.568181 15929 ts_tablet_manager.cc:1918] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:33.568368 15929 log.cc:1196] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/8fdcce97cf26424498a14bd803977f87
I20260806 11:42:33.568790 15929 ts_tablet_manager.cc:1939] T 8fdcce97cf26424498a14bd803977f87 P 89c76f29b2f64f57a522e0ee1c7cbed3: Deleting consensus metadata
I20260806 11:42:33.570150 15150 catalog_manager.cc:4841] TS 89c76f29b2f64f57a522e0ee1c7cbed3 (127.0.113.3:45405): tablet 8fdcce97cf26424498a14bd803977f87 (table test-table [id=a386d3513678480e917538eaf4717633]) successfully deleted
I20260806 11:42:33.570492 15928 ts_tablet_manager.cc:1918] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:42:33.570606 15928 log.cc:1196] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/8fdcce97cf26424498a14bd803977f87
I20260806 11:42:33.570984 15928 ts_tablet_manager.cc:1939] T 8fdcce97cf26424498a14bd803977f87 P d85e5fbc53a04502a4975c4334358cc0: Deleting consensus metadata
W20260806 11:42:33.571525 15150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:38665 (error 108)
I20260806 11:42:33.572026 15150 catalog_manager.cc:4841] TS d85e5fbc53a04502a4975c4334358cc0 (127.0.113.2:45651): tablet 8fdcce97cf26424498a14bd803977f87 (table test-table [id=a386d3513678480e917538eaf4717633]) successfully deleted
I20260806 11:42:33.572589 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 15490
I20260806 11:42:33.572638 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:42:33.572345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:38665 (local address 127.0.0.1:34478)
0806 11:42:33.572479 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 11:42:33.572552 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:38665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 11:42:33.578472 15150 catalog_manager.cc:4584] TS e9f2fa8358f6457aba4fda49885e3943 (127.0.113.1:38665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fdcce97cf26424498a14bd803977f87: Network error: Client connection negotiation failed: client connection to 127.0.113.1:38665: connect: Connection refused (error 111)
I20260806 11:42:33.586606 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 15686
I20260806 11:42:33.597271 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 15130
2026-08-06T11:42:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2694 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:42:33 dist-test-slave-45dx krb5kdc[15935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:42:33 dist-test-slave-45dx krb5kdc[15935](info): set up 2 sockets
Aug 06 11:42:33 dist-test-slave-45dx krb5kdc[15935](info): commencing operation
krb5kdc: starting...
W20260806 11:42:34.823917 452 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:42:34 dist-test-slave-45dx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:42:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/rangeradmin_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/rangeradmin_127.0.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/rangerlookup_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/rangerlookup_127.0.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/HTTP_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/HTTP_127.0.113.20@KRBTEST.COM.keytab.
I20260806 11:42:35.001266 452 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskg6qW6_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/postgres -l logfile start
2026-08-06 11:42:37.337 UTC [15974] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 11:42:37.337 UTC [15974] LOG: listening on IPv4 address "127.0.113.20", port 46465
2026-08-06 11:42:37.340 UTC [15974] LOG: listening on Unix socket "/tmp/.s.PGSQL.46465"
2026-08-06 11:42:37.362 UTC [15980] LOG: database system was shut down at 2026-08-06 11:42:36 UTC
2026-08-06 11:42:37.365 UTC [15974] LOG: database system is ready to accept connections
I20260806 11:42:38.507093 452 mini_postgres.cc:96] Postgres bound to 46465
2026-08-06 11:42:38.514 UTC [15990] FATAL: database "slave" does not exist
127.0.113.20:46465 - accepting connections
I20260806 11:42:38.515661 452 mini_ranger.cc:161] Starting Ranger...
I20260806 11:42:38.531529 452 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 11:42:38.991969 452 mini_ranger.cc:87] Created ranger Postgres database
I20260806 11:42:38.992091 452 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-admin
2026-08-06 11:42:39,302 [I] DB FLAVOR :POSTGRES
2026-08-06 11:42:39,302 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 11:42:39,302 [I] Checking connection..
2026-08-06 11:42:39,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 11:42:39,564 [I] Checking connection passed.
2026-08-06 11:42:39,564 [I] --------- Verifying version history table ---------
2026-08-06 11:42:39,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 11:42:39,804 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 11:42:39,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 11:42:40,072 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 11:42:40,072 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 11:42:40,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 11:42:40.303 UTC [16087] WARNING: there is no transaction in progress
2026-08-06 11:42:40,312 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 11:42:40,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 11:42:40,563 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 11:42:40,563 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 11:42:40,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 11:42:40,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 11:42:41,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-45dx.c.gcp-upstream.internal','N') ;"
2026-08-06 11:42:41,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 11:42:41,537 [I] Table x_portal_user does not exist in database ranger
2026-08-06 11:42:41,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 11:42:41,803 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 11:42:41,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 11:42:42,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 11:42:42,310 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 11:42:42,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 11:42:43.244 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.259 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.275 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.289 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.307 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.325 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.381 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.388 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.402 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.412 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.423 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.436 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.444 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.450 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.456 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:43.462 UTC [16236] WARNING: there is no transaction in progress
2026-08-06 11:42:44,081 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 11:42:44,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 11:42:44,335 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 11:42:44,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 11:42:44,584 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 11:42:44,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 11:42:44,834 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 11:42:44,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 11:42:45,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 11:42:45,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-45dx.c.gcp-upstream.internal';"
2026-08-06 11:42:45,596 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 11:42:45,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 11:42:45,856 [I] DB_PATCHES have already been applied
I20260806 11:42:45.862047 452 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-admin:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskg6qW6_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 11:42:45.862116 452 mini_ranger.cc:193] Using host: 127.0.113.20
I20260806 11:42:45.866721 452 mini_ranger.cc:227] Ranger admin URL: http://127.0.113.20:38725
Aug 06, 2026 11:42:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:42:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:42:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 11:42:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 11:42:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 11:42:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 11:42:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 11:42:47.169621 452 mini_ranger.cc:160] Time spent starting Ranger: real 8.654s user 0.000s sys 0.013s
Aug 06, 2026 11:43:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 11:43:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 11:43:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 11:43:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 11:43:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 11:43:08.231849 452 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/rangerkms_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/rangerkms_127.0.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.113.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.113.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.113.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/HTTP_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.113.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/HTTP_127.0.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 11:43:08 dist-test-slave-45dx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016588, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 11:43:08.322533 452 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 11:43:08.333063 452 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 11:43:09.215124 452 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 11:43:09.215214 452 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-kms
I20260806 11:43:09.215317 452 mini_ranger_kms.cc:227] Using postgres at 127.0.113.20:46465
2026-08-06 11:43:09,572 [I] DB FLAVOR :POSTGRES
2026-08-06 11:43:09,572 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 11:43:09,572 [I] Checking connection
2026-08-06 11:43:09,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 11:43:09,858 [I] connection success
2026-08-06 11:43:09,859 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 11:43:09,859 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 11:43:09,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 11:43:10,104 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 11:43:10,104 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 11:43:10,104 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 11:43:10,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:46465/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 11:43:10,379 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 11:43:10.852748 452 mini_ranger_kms.cc:310] Created kms service
I20260806 11:43:11.082264 452 mini_ranger_kms.cc:326] Created kudu user
I20260806 11:43:11.127252 452 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 11:43:11 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016588, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:43:11.525317 452 mini_ranger_kms.cc:384] Added ranger policy
I20260806 11:43:11.525496 452 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-kms:/tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 11:43:11.525525 452 mini_ranger_kms.cc:240] Using host: 127.0.113.20
I20260806 11:43:11.530894 452 mini_ranger_kms.cc:276] Ranger KMS PID: 16499
I20260806 11:43:11.531032 452 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.113.20:36265
Aug 06, 2026 11:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 11:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 11:43:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 11:43:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 11:43:13 dist-test-slave-45dx krb5kdc[15935](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786016593, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.113.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:43:14.242969 452 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.920s user 0.020s sys 0.316s
I20260806 11:43:14.243115 452 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 11:43:14.243239 452 mini_ranger_kms.cc:401] 127.0.113.20:36265/kms/v1/keys
Aug 06 11:43:14 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016593, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.113.20@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
Aug 06 11:43:14 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016593, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.113.20@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
Aug 06 11:43:17 dist-test-slave-45dx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:43:17.181931 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33657
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:33657
--ranger_config_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:36265/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:43:17.304481 16575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:43:17.304728 16575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:43:17.304792 16575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:43:17.308928 16575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:43:17.308984 16575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:43:17.309003 16575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:43:17.309018 16575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:43:17.309053 16575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:43:17.314275 16575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.113.20:36265/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:33657
--ranger_config_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:43:17.315249 16575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:43:17.316515 16575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:43:17.324949 16585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:43:17.325031 16575 server_base.cc:1034] running on GCE node
W20260806 11:43:17.325160 16581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:43:17.325006 16580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:43:17.325917 16575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:43:17.326926 16575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:43:17.328094 16575 hybrid_clock.cc:648] HybridClock initialized: now 1786016597328070 us; error 37 us; skew 500 ppm
Aug 06 11:43:17 dist-test-slave-45dx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:43:17.334968 16575 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:43:17.335886 16575 webserver.cc:460] Webserver started at http://127.0.113.62:38781/ using document root <none> and password file <none>
I20260806 11:43:17.336481 16575 fs_manager.cc:359] Metadata directory not provided
I20260806 11:43:17.336560 16575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:43:17.336735 16575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 11:43:17 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:43:17.384434 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "1c8556106bff47f1b77d28774d5b5c58"
format_stamp: "Formatted at 2026-08-06 11:43:17 on dist-test-slave-45dx"
server_key: "ce6dbe5fd8f7f3c5de09c675da345873"
server_key_iv: "495384e73f11ef6cc52d5315210adac4"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:17.385131 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "1c8556106bff47f1b77d28774d5b5c58"
format_stamp: "Formatted at 2026-08-06 11:43:17 on dist-test-slave-45dx"
server_key: "ce6dbe5fd8f7f3c5de09c675da345873"
server_key_iv: "495384e73f11ef6cc52d5315210adac4"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:17.389407 16575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 11:43:17.405072 16595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:17.406639 16575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:43:17.406795 16575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "1c8556106bff47f1b77d28774d5b5c58"
format_stamp: "Formatted at 2026-08-06 11:43:17 on dist-test-slave-45dx"
server_key: "ce6dbe5fd8f7f3c5de09c675da345873"
server_key_iv: "495384e73f11ef6cc52d5315210adac4"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:17.406941 16575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:43:17.419456 16575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:43:17.422154 16575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:43:17.422317 16575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:43:17.431847 16575 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:33657
I20260806 11:43:17.431882 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:33657 every 8 connection(s)
I20260806 11:43:17.432811 16575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:43:17.435583 16672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:43:17.439287 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 16575
I20260806 11:43:17.439440 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:43:17.439697 452 external_mini_cluster.cc:1428] Setting key e4479475f2ddd9eff423ec5ff01e7259
I20260806 11:43:17.441983 16672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58: Bootstrap starting.
I20260806 11:43:17.444613 16672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58: Neither blocks nor log segments found. Creating new log.
I20260806 11:43:17.445465 16672 log.cc:824] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58: Log is configured to *not* fsync() on all Append() calls
I20260806 11:43:17.448009 16672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58: No bootstrap required, opened a new log
Aug 06 11:43:17 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:43:17.453187 16672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8556106bff47f1b77d28774d5b5c58" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33657 } }
I20260806 11:43:17.453469 16672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:43:17.453528 16672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8556106bff47f1b77d28774d5b5c58, State: Initialized, Role: FOLLOWER
I20260806 11:43:17.454010 16672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8556106bff47f1b77d28774d5b5c58" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33657 } }
I20260806 11:43:17.454173 16672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:43:17.454241 16672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:43:17.454329 16672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:43:17.455333 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:17.440981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.0.113.62:33657)
0806 11:43:17.441351 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:17.441365 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:17.441400 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:43:17.442122 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:17.442146 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:17.442466 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:17.442732 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:43:17.443666 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:17.445107 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:17.446348 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:17.447043 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:17.447199 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:17.450206 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:43:17.450212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:43:17.450230 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:43:17.450279 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:43:17.452296 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:17.452732 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:17.452738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:17.452744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:17.452890 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:17.453153 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:17.453156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:17.453157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:17.453657 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:43:17.453867 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:17.454136 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 11:43:17.454407 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":118,"threads_started":1}
I20260806 11:43:17.455385 16672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8556106bff47f1b77d28774d5b5c58" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33657 } }
I20260806 11:43:17.455735 16672 leader_election.cc:302] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c8556106bff47f1b77d28774d5b5c58; no voters:
I20260806 11:43:17.456043 16672 leader_election.cc:288] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:43:17.456254 16684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:43:17.456574 16684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [term 1 LEADER]: Becoming Leader. State: Replica: 1c8556106bff47f1b77d28774d5b5c58, State: Running, Role: LEADER
I20260806 11:43:17.456849 16684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8556106bff47f1b77d28774d5b5c58" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33657 } }
I20260806 11:43:17.457222 16672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:43:17.458035 16684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c8556106bff47f1b77d28774d5b5c58. Latest consensus state: current_term: 1 leader_uuid: "1c8556106bff47f1b77d28774d5b5c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8556106bff47f1b77d28774d5b5c58" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33657 } } }
I20260806 11:43:17.458154 16684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [sys.catalog]: This master's current role is: LEADER
I20260806 11:43:17.460062 16686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c8556106bff47f1b77d28774d5b5c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8556106bff47f1b77d28774d5b5c58" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33657 } } }
I20260806 11:43:17.460165 16686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58 [sys.catalog]: This master's current role is: LEADER
I20260806 11:43:17.460871 16694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:43:17.464196 16694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:43:17.464779 16672 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 11:43:17.476147 16694 catalog_manager.cc:1347] Generated new cluster ID: 5d930537d0ae491c81e4a492b9e9ffb0
I20260806 11:43:17.476228 16694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:43:17.490043 16694 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:43:17.490916 16694 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:43:17.513913 16694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58: Generated new TSK 0
I20260806 11:43:17.514613 16694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:43:18.335676 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:36265/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33657
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:43:18.506678 16740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:43:18.506932 16740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:43:18.507001 16740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:43:18.512161 16740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:43:18.512279 16740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:43:18.512406 16740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:43:18.517427 16740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.113.20:36265/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:43:18.518389 16740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:43:18.519340 16740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:43:18.530476 16749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:43:18.530428 16750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:43:18.530485 16752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:43:18.530813 16740 server_base.cc:1034] running on GCE node
I20260806 11:43:18.531177 16740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:43:18.531805 16740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:43:18.532980 16740 hybrid_clock.cc:648] HybridClock initialized: now 1786016598532965 us; error 37 us; skew 500 ppm
Aug 06 11:43:18 dist-test-slave-45dx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:43:18.536183 16740 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:43:18.537318 16740 webserver.cc:460] Webserver started at http://127.0.113.1:40041/ using document root <none> and password file <none>
I20260806 11:43:18.538002 16740 fs_manager.cc:359] Metadata directory not provided
I20260806 11:43:18.538103 16740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:43:18.538349 16740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 11:43:18 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:43:18.555088 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "4ed5f31f5c924d0595b0ced91d9bc89d"
format_stamp: "Formatted at 2026-08-06 11:43:18 on dist-test-slave-45dx"
server_key: "f1f60aae4e7fe4324e5970f4fbb7b823"
server_key_iv: "2870b5afe18a2977708ae3097de2f3cb"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:18.556051 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "4ed5f31f5c924d0595b0ced91d9bc89d"
format_stamp: "Formatted at 2026-08-06 11:43:18 on dist-test-slave-45dx"
server_key: "f1f60aae4e7fe4324e5970f4fbb7b823"
server_key_iv: "2870b5afe18a2977708ae3097de2f3cb"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:18.560621 16740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 11:43:18.577345 16762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:18.579492 16740 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 11:43:18.579674 16740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "4ed5f31f5c924d0595b0ced91d9bc89d"
format_stamp: "Formatted at 2026-08-06 11:43:18 on dist-test-slave-45dx"
server_key: "f1f60aae4e7fe4324e5970f4fbb7b823"
server_key_iv: "2870b5afe18a2977708ae3097de2f3cb"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:18.579815 16740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:43:18.616227 16740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:43:18.619346 16740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:43:18.619558 16740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:43:18.620271 16740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:43:18.621182 16740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:43:18.621250 16740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:18.621315 16740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:43:18.621359 16740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:18.633914 16740 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:42873
I20260806 11:43:18.635087 16740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:43:18.643998 16934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:42873 every 8 connection(s)
I20260806 11:43:18.645371 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 16740
I20260806 11:43:18.645524 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:43:18.645846 452 external_mini_cluster.cc:1428] Setting key dbdc20846455ce1864735aded19d9209
Aug 06 11:43:18 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:43:18.665288 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:18.646386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33657 (local address 127.0.113.1:47917)
0806 11:43:18.649057 (+ 2671us) negotiation.cc:107] Waiting for socket to connect
0806 11:43:18.649115 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 11:43:18.650560 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:43:18.651402 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:43:18.651412 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:43:18.651830 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:43:18.652573 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:18.652591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:18.653718 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:18.653724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:18.655291 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:18.655305 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:18.656194 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:18.656200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:18.656461 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:18.657235 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:43:18.657257 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:43:18.659132 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:43:18.661923 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:43:18.661929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:43:18.661944 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:43:18.662198 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:43:18.662499 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:43:18.662502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:43:18.662504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:43:18.662679 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:43:18.663196 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:43:18.663203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:43:18.663551 (+ 348us) client_negotiation.cc:769] Sending connection context
0806 11:43:18.663757 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 11:43:18.664110 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2433,"thread_start_us":175,"threads_started":1}
I20260806 11:43:18.666137 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:18.647730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:47917 (local address 127.0.113.62:33657)
0806 11:43:18.649911 (+ 2181us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:18.649916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:18.649930 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:43:18.650900 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:18.650905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:18.650965 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:18.651068 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:43:18.652717 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:18.653565 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:18.655446 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:18.655892 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:18.656007 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:18.659362 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:43:18.659367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:43:18.659370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:43:18.659399 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:43:18.661768 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:18.662309 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:18.662312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:18.662314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:18.662405 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:18.662788 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:18.662791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:18.662793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:18.663010 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:43:18.663136 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:18.665745 (+ 2609us) server_negotiation.cc:299] Negotiation successful
0806 11:43:18.665901 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2106,"thread_start_us":106,"threads_started":1}
I20260806 11:43:18.667269 16938 heartbeater.cc:347] Connected to a master server at 127.0.113.62:33657
I20260806 11:43:18.667573 16938 heartbeater.cc:464] Registering TS with master...
I20260806 11:43:18.668071 16938 heartbeater.cc:511] Master 127.0.113.62:33657 requested a full tablet report, sending...
I20260806 11:43:18.669806 16619 ts_manager.cc:194] Registered new tserver with Master: 4ed5f31f5c924d0595b0ced91d9bc89d (127.0.113.1:42873)
I20260806 11:43:18.671437 16619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:47917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:43:18.685997 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:36265/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33657
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:43:18 dist-test-slave-45dx krb5kdc[15935](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786016598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 11:43:18.825639 16946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:43:18.825956 16946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:43:18.826030 16946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:43:18.830209 16946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:43:18.830294 16946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:43:18.830391 16946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:43:18.836382 16946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.113.20:36265/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:43:18.837504 16946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:43:18.838593 16946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:43:18.849869 16955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:43:18.849925 16954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:43:18.850314 16958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:43:18.850293 16946 server_base.cc:1034] running on GCE node
I20260806 11:43:18.850852 16946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:43:18.851724 16946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:43:18.852942 16946 hybrid_clock.cc:648] HybridClock initialized: now 1786016598852904 us; error 44 us; skew 500 ppm
Aug 06 11:43:18 dist-test-slave-45dx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:43:18.856504 16946 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:43:18.857515 16946 webserver.cc:460] Webserver started at http://127.0.113.2:43545/ using document root <none> and password file <none>
I20260806 11:43:18.858163 16946 fs_manager.cc:359] Metadata directory not provided
I20260806 11:43:18.858229 16946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:43:18.858491 16946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 11:43:18 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:43:18.874332 16946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "44ce62e5e021428897b8c046a00f4a0e"
format_stamp: "Formatted at 2026-08-06 11:43:18 on dist-test-slave-45dx"
server_key: "bcc810638b2c8e366f70d6019365f342"
server_key_iv: "0c39f20d82696e2b7c1a2d516f4a08b1"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:18.875103 16946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "44ce62e5e021428897b8c046a00f4a0e"
format_stamp: "Formatted at 2026-08-06 11:43:18 on dist-test-slave-45dx"
server_key: "bcc810638b2c8e366f70d6019365f342"
server_key_iv: "0c39f20d82696e2b7c1a2d516f4a08b1"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:18.886791 16946 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.003s sys 0.005s
I20260806 11:43:18.911906 16968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:18.913651 16946 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.001s
I20260806 11:43:18.913803 16946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "44ce62e5e021428897b8c046a00f4a0e"
format_stamp: "Formatted at 2026-08-06 11:43:18 on dist-test-slave-45dx"
server_key: "bcc810638b2c8e366f70d6019365f342"
server_key_iv: "0c39f20d82696e2b7c1a2d516f4a08b1"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:18.913949 16946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:43:18.934538 16946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:43:18.938354 16946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:43:18.938547 16946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:43:18.939321 16946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:43:18.940485 16946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:43:18.940550 16946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:18.940601 16946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:43:18.940630 16946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:18.963131 16946 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:36283
I20260806 11:43:18.964358 16946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:43:18.966601 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 16946
I20260806 11:43:18.966740 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:43:18.967077 452 external_mini_cluster.cc:1428] Setting key 96e23a49a106a41c455afc2bb94fd968
I20260806 11:43:18.976202 17138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:36283 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Aug 06 11:43:18 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:43:19.008697 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:18.974260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:56699 (local address 127.0.113.62:33657)
0806 11:43:18.974529 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:18.974534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:18.980883 (+ 6349us) server_negotiation.cc:407] Connection header received
0806 11:43:18.982548 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:18.982552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:18.982612 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:18.982700 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:43:18.993697 (+ 10997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:18.994686 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:18.999697 (+ 5011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.000091 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.000181 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.003617 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:43:19.003621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:43:19.003624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:43:19.003658 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:43:19.005751 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:19.006501 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:19.006505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:19.006507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:19.006603 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:19.007098 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:19.007102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:19.007103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:19.007441 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:43:19.007553 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:19.008337 (+ 784us) server_negotiation.cc:299] Negotiation successful
0806 11:43:19.008492 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260806 11:43:19.011425 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:18.974979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33657 (local address 127.0.113.2:56699)
0806 11:43:18.980666 (+ 5687us) negotiation.cc:107] Waiting for socket to connect
0806 11:43:18.980729 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 11:43:18.982302 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:43:18.983075 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:43:18.983086 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:43:18.983625 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:43:18.984582 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:18.984609 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:18.995128 (+ 10519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:18.995132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:18.999522 (+ 4390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:18.999538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.000458 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.000463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.000690 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.001438 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:43:19.001460 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:43:19.003384 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:43:19.005957 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:43:19.005964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:43:19.005982 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:43:19.006370 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:43:19.006726 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:43:19.006729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:43:19.006731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:43:19.006977 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:43:19.007632 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:43:19.007639 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:43:19.008015 (+ 376us) client_negotiation.cc:769] Sending connection context
0806 11:43:19.009208 (+ 1193us) client_negotiation.cc:240] Negotiation successful
0806 11:43:19.009580 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5431,"thread_start_us":233,"threads_started":1}
I20260806 11:43:19.013724 17139 heartbeater.cc:347] Connected to a master server at 127.0.113.62:33657
I20260806 11:43:19.013964 17139 heartbeater.cc:464] Registering TS with master...
I20260806 11:43:19.014576 17139 heartbeater.cc:511] Master 127.0.113.62:33657 requested a full tablet report, sending...
I20260806 11:43:19.015822 16619 ts_manager.cc:194] Registered new tserver with Master: 44ce62e5e021428897b8c046a00f4a0e (127.0.113.2:36283)
I20260806 11:43:19.016645 16619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:56699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:43:19.040704 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:36265/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33657
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:43:19.200505 17145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:43:19.200970 17145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:43:19.201102 17145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:43:19.207165 17145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:43:19.207310 17145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:43:19.207450 17145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:43:19.215461 17145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.113.20:36265/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:43:19.217267 17145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:43:19.218885 17145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 11:43:19.232919 17145 server_base.cc:1034] running on GCE node
W20260806 11:43:19.233150 17153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:43:19.233428 17151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:43:19.233844 17157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:43:19.234367 17145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:43:19.235123 17145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:43:19.236330 17145 hybrid_clock.cc:648] HybridClock initialized: now 1786016599236246 us; error 79 us; skew 500 ppm
Aug 06 11:43:19 dist-test-slave-45dx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:43:19.240101 17145 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:43:19.241222 17145 webserver.cc:460] Webserver started at http://127.0.113.3:41025/ using document root <none> and password file <none>
I20260806 11:43:19.242002 17145 fs_manager.cc:359] Metadata directory not provided
I20260806 11:43:19.242146 17145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:43:19.242434 17145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 11:43:19 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:43:19.267932 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "5e7a8704427b45a3b6327528df991b24"
format_stamp: "Formatted at 2026-08-06 11:43:19 on dist-test-slave-45dx"
server_key: "59ce742e7d6eed4cead81fdbe108d661"
server_key_iv: "e63aa37e0a852965e4a3f635fcf8531a"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:19.276819 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "5e7a8704427b45a3b6327528df991b24"
format_stamp: "Formatted at 2026-08-06 11:43:19 on dist-test-slave-45dx"
server_key: "59ce742e7d6eed4cead81fdbe108d661"
server_key_iv: "e63aa37e0a852965e4a3f635fcf8531a"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:19.286726 17145 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.002s
Aug 06 11:43:19 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:43:19.308655 17165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:19.310288 17145 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260806 11:43:19.310418 17145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "5e7a8704427b45a3b6327528df991b24"
format_stamp: "Formatted at 2026-08-06 11:43:19 on dist-test-slave-45dx"
server_key: "59ce742e7d6eed4cead81fdbe108d661"
server_key_iv: "e63aa37e0a852965e4a3f635fcf8531a"
server_key_version: "kuduclusterkey@0"
I20260806 11:43:19.310540 17145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:43:19.326591 17145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:43:19.331197 17145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:43:19.331372 17145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:43:19.331964 17145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:43:19.333287 17145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:43:19.333348 17145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:19.333400 17145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:43:19.333429 17145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:43:19.345005 17145 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:45741
I20260806 11:43:19.346364 17145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:43:19.352779 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 17145
I20260806 11:43:19.352913 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:43:19.353276 452 external_mini_cluster.cc:1428] Setting key 73e45e045744c766c0f235f1cb22fc4b
I20260806 11:43:19.355729 17333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:45741 every 8 connection(s)
Aug 06 11:43:19 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:43:19.375291 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.359165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:55829 (local address 127.0.113.62:33657)
0806 11:43:19.359279 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:19.359284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:19.359949 (+ 665us) server_negotiation.cc:407] Connection header received
0806 11:43:19.361201 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:19.361204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:19.361257 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:19.361303 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:43:19.362989 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.363810 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.365552 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.365951 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.366033 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.370942 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:43:19.370946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:43:19.370949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:43:19.370977 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:43:19.372749 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:19.373394 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:19.373398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:19.373400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:19.373507 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:19.373941 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:19.373944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:19.373946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:19.374163 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:43:19.374249 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:19.375000 (+ 751us) server_negotiation.cc:299] Negotiation successful
0806 11:43:19.375123 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 11:43:19.377111 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.358111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33657 (local address 127.0.113.3:55829)
0806 11:43:19.359764 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0806 11:43:19.359818 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 11:43:19.360924 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:43:19.361695 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:43:19.361704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:43:19.362122 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:43:19.362785 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.362805 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.363983 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.363986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.365404 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.365416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.367762 (+ 2346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.367766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.367999 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.368765 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:43:19.368786 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:43:19.370742 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:43:19.372922 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:43:19.372928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:43:19.372943 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:43:19.373275 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:43:19.373615 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:43:19.373618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:43:19.373620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:43:19.373807 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:43:19.374320 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:43:19.374326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:43:19.374650 (+ 324us) client_negotiation.cc:769] Sending connection context
0806 11:43:19.375726 (+ 1076us) client_negotiation.cc:240] Negotiation successful
0806 11:43:19.376089 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"thread_start_us":149,"threads_started":1}
I20260806 11:43:19.379688 17334 heartbeater.cc:347] Connected to a master server at 127.0.113.62:33657
I20260806 11:43:19.379920 17334 heartbeater.cc:464] Registering TS with master...
I20260806 11:43:19.380501 17334 heartbeater.cc:511] Master 127.0.113.62:33657 requested a full tablet report, sending...
I20260806 11:43:19.382535 16619 ts_manager.cc:194] Registered new tserver with Master: 5e7a8704427b45a3b6327528df991b24 (127.0.113.3:45741)
I20260806 11:43:19.383266 16619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:55829
I20260806 11:43:19.393463 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:43:19.429502 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.400729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.0.113.62:33657)
0806 11:43:19.400840 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:19.400845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:19.410177 (+ 9332us) server_negotiation.cc:407] Connection header received
0806 11:43:19.410439 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:19.410443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:19.410494 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:19.410548 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:43:19.411654 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.412454 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.413676 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.414004 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.414076 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.425277 (+ 11201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:43:19.425282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:43:19.425285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:43:19.425320 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:43:19.427241 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:19.427962 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:19.427966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:19.427968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:19.428065 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:43:19.428431 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:43:19.428434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:43:19.428436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:43:19.428695 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:43:19.428805 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:19.429181 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 11:43:19.429312 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 11:43:19.440027 16619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:43:19.444456 16619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:43:19.478359 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.465469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:45741 (local address 127.0.0.1:36610)
0806 11:43:19.468629 (+ 3160us) negotiation.cc:107] Waiting for socket to connect
0806 11:43:19.468685 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 11:43:19.468823 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:43:19.473235 (+ 4412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:43:19.473241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:43:19.473264 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:43:19.473436 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.473443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.474927 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.474930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.476287 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.476340 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.477700 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.477704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.477923 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.477965 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 11:43:19.478073 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 11:43:19.478186 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2929,"thread_start_us":130,"threads_started":1}
I20260806 11:43:19.479053 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.466596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.0.113.3:45741)
0806 11:43:19.472542 (+ 5946us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:19.472559 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:19.472586 (+ 27us) server_negotiation.cc:407] Connection header received
0806 11:43:19.472665 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:19.472671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:19.472891 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:19.473055 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:43:19.473583 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.474781 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.476497 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.477362 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.477469 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.477636 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:19.478776 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0806 11:43:19.478887 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5829,"thread_start_us":175,"threads_started":1}
I20260806 11:43:19.480798 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.467253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:36283 (local address 127.0.0.1:37286)
0806 11:43:19.472523 (+ 5270us) negotiation.cc:107] Waiting for socket to connect
0806 11:43:19.472539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:43:19.472627 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:43:19.473643 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:43:19.473646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:43:19.473664 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:43:19.473751 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.473756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.476140 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.476144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.477323 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.477345 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.480402 (+ 3057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.480406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.480500 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.480524 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 11:43:19.480589 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 11:43:19.480652 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5198,"thread_start_us":111,"threads_started":1}
I20260806 11:43:19.482440 17221 tablet_service.cc:1467] Processing CreateTablet for tablet d35faf3b8b2d4e1aaf339951d3a8650e (DEFAULT_TABLE table=test-table [id=86191ceba2a147af86a78b40563a1f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:43:19.483690 17221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35faf3b8b2d4e1aaf339951d3a8650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:43:19.494120 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.467824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.0.113.2:36283)
0806 11:43:19.472870 (+ 5046us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:19.472888 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:19.472912 (+ 24us) server_negotiation.cc:407] Connection header received
0806 11:43:19.472988 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:19.472993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:19.473306 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:19.473479 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:43:19.474153 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.476018 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.478502 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.479424 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.479531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.493721 (+ 14190us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:19.493787 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 11:43:19.493905 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4935,"thread_start_us":133,"threads_started":1}
I20260806 11:43:19.497363 17041 tablet_service.cc:1467] Processing CreateTablet for tablet d35faf3b8b2d4e1aaf339951d3a8650e (DEFAULT_TABLE table=test-table [id=86191ceba2a147af86a78b40563a1f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:43:19.498523 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35faf3b8b2d4e1aaf339951d3a8650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:43:19.508838 17364 tablet_bootstrap.cc:492] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: Bootstrap starting.
I20260806 11:43:19.513053 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.469197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:42873 (local address 127.0.0.1:52766)
0806 11:43:19.473952 (+ 4755us) negotiation.cc:107] Waiting for socket to connect
0806 11:43:19.473964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:43:19.474044 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:43:19.478220 (+ 4176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:43:19.478224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:43:19.478241 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:43:19.478340 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.478346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.482431 (+ 4085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.482434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.492569 (+ 10135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.492585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.512582 (+ 19997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.512586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.512697 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.512725 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 11:43:19.512806 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 11:43:19.512903 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4698,"thread_start_us":104,"threads_started":1}
I20260806 11:43:19.513727 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.469637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.0.113.1:42873)
0806 11:43:19.477636 (+ 7999us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:19.477642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:19.477664 (+ 22us) server_negotiation.cc:407] Connection header received
0806 11:43:19.477736 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:19.477741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:19.477925 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:19.478070 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:43:19.481083 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.482258 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.511180 (+ 28922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.512130 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.512271 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.512490 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:19.513484 (+ 994us) server_negotiation.cc:299] Negotiation successful
0806 11:43:19.513593 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7889,"thread_start_us":73,"threads_started":1}
I20260806 11:43:19.516836 17364 tablet_bootstrap.cc:654] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: Neither blocks nor log segments found. Creating new log.
I20260806 11:43:19.517313 16834 tablet_service.cc:1467] Processing CreateTablet for tablet d35faf3b8b2d4e1aaf339951d3a8650e (DEFAULT_TABLE table=test-table [id=86191ceba2a147af86a78b40563a1f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:43:19.517853 17364 log.cc:824] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: Log is configured to *not* fsync() on all Append() calls
I20260806 11:43:19.519624 16834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35faf3b8b2d4e1aaf339951d3a8650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:43:19.530212 17364 tablet_bootstrap.cc:492] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: No bootstrap required, opened a new log
I20260806 11:43:19.530427 17364 ts_tablet_manager.cc:1397] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20260806 11:43:19.531772 17370 tablet_bootstrap.cc:492] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: Bootstrap starting.
I20260806 11:43:19.534166 17370 tablet_bootstrap.cc:654] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: Neither blocks nor log segments found. Creating new log.
I20260806 11:43:19.534605 17369 tablet_bootstrap.cc:492] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d: Bootstrap starting.
I20260806 11:43:19.534433 17364 raft_consensus.cc:348] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } }
I20260806 11:43:19.534716 17364 raft_consensus.cc:374] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:43:19.534770 17364 raft_consensus.cc:729] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7a8704427b45a3b6327528df991b24, State: Initialized, Role: FOLLOWER
I20260806 11:43:19.535279 17364 consensus_queue.cc:260] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } }
I20260806 11:43:19.536330 17364 ts_tablet_manager.cc:1428] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260806 11:43:19.536707 17334 heartbeater.cc:503] Master 127.0.113.62:33657 was elected leader, sending a full tablet report...
I20260806 11:43:19.537505 17369 tablet_bootstrap.cc:654] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d: Neither blocks nor log segments found. Creating new log.
I20260806 11:43:19.538148 17370 log.cc:824] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: Log is configured to *not* fsync() on all Append() calls
I20260806 11:43:19.539032 17369 log.cc:824] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d: Log is configured to *not* fsync() on all Append() calls
I20260806 11:43:19.541657 17370 tablet_bootstrap.cc:492] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: No bootstrap required, opened a new log
I20260806 11:43:19.541972 17370 ts_tablet_manager.cc:1397] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260806 11:43:19.547910 17369 tablet_bootstrap.cc:492] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d: No bootstrap required, opened a new log
I20260806 11:43:19.548141 17369 ts_tablet_manager.cc:1397] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260806 11:43:19.549628 17370 raft_consensus.cc:348] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } }
I20260806 11:43:19.550021 17370 raft_consensus.cc:374] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:43:19.550129 17370 raft_consensus.cc:729] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ce62e5e021428897b8c046a00f4a0e, State: Initialized, Role: FOLLOWER
I20260806 11:43:19.550694 17370 consensus_queue.cc:260] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } }
I20260806 11:43:19.551998 17370 ts_tablet_manager.cc:1428] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260806 11:43:19.552441 17139 heartbeater.cc:503] Master 127.0.113.62:33657 was elected leader, sending a full tablet report...
I20260806 11:43:19.552757 17369 raft_consensus.cc:348] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } }
I20260806 11:43:19.552994 17369 raft_consensus.cc:374] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:43:19.553051 17369 raft_consensus.cc:729] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed5f31f5c924d0595b0ced91d9bc89d, State: Initialized, Role: FOLLOWER
I20260806 11:43:19.553494 17369 consensus_queue.cc:260] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } }
I20260806 11:43:19.554569 17369 ts_tablet_manager.cc:1428] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 11:43:19.554917 16938 heartbeater.cc:503] Master 127.0.113.62:33657 was elected leader, sending a full tablet report...
W20260806 11:43:19.609869 17338 tablet.cc:2369] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:43:19.645493 16940 tablet.cc:2369] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 11:43:19 dist-test-slave-45dx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:43:19.707988 17372 raft_consensus.cc:482] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:43:19.708354 17372 raft_consensus.cc:504] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } }
I20260806 11:43:19.709898 17372 leader_election.cc:288] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ce62e5e021428897b8c046a00f4a0e (127.0.113.2:36283), 4ed5f31f5c924d0595b0ced91d9bc89d (127.0.113.1:42873)
I20260806 11:43:19.716516 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.710273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:36283 (local address 127.0.113.3:37227)
0806 11:43:19.711035 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0806 11:43:19.711050 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:43:19.711137 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:43:19.712291 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:43:19.712295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:43:19.712334 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:43:19.712591 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.712599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.713598 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.713601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.714850 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.714863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.715963 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.715967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.716172 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.716200 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 11:43:19.716272 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 11:43:19.716362 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":17}
I20260806 11:43:19.717007 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.711429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:37227 (local address 127.0.113.2:36283)
0806 11:43:19.711573 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:19.711578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:19.711594 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:43:19.711651 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:19.711654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:19.711703 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:19.711783 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:43:19.712735 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.713444 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.715080 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.715690 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.715791 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.715884 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:19.716776 (+ 892us) server_negotiation.cc:299] Negotiation successful
0806 11:43:19.716880 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 11:43:19.717886 17050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35faf3b8b2d4e1aaf339951d3a8650e" candidate_uuid: "5e7a8704427b45a3b6327528df991b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ce62e5e021428897b8c046a00f4a0e" is_pre_election: true
I20260806 11:43:19.718221 17050 raft_consensus.cc:2454] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7a8704427b45a3b6327528df991b24 in term 0.
I20260806 11:43:19.718833 17171 leader_election.cc:302] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ce62e5e021428897b8c046a00f4a0e, 5e7a8704427b45a3b6327528df991b24; no voters:
I20260806 11:43:19.719123 17372 raft_consensus.cc:2781] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:43:19.719238 17372 raft_consensus.cc:482] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:43:19.719291 17372 raft_consensus.cc:3037] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:43:19.721096 17372 raft_consensus.cc:504] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } }
I20260806 11:43:19.721472 17372 leader_election.cc:288] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [CANDIDATE]: Term 1 election: Requested vote from peers 44ce62e5e021428897b8c046a00f4a0e (127.0.113.2:36283), 4ed5f31f5c924d0595b0ced91d9bc89d (127.0.113.1:42873)
I20260806 11:43:19.721844 17050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35faf3b8b2d4e1aaf339951d3a8650e" candidate_uuid: "5e7a8704427b45a3b6327528df991b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ce62e5e021428897b8c046a00f4a0e"
I20260806 11:43:19.721992 17050 raft_consensus.cc:3037] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:43:19.723325 17050 raft_consensus.cc:2454] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7a8704427b45a3b6327528df991b24 in term 1.
W20260806 11:43:19.723480 17140 tablet.cc:2369] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:43:19.723623 17171 leader_election.cc:302] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ce62e5e021428897b8c046a00f4a0e, 5e7a8704427b45a3b6327528df991b24; no voters:
I20260806 11:43:19.723767 17372 raft_consensus.cc:2781] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:43:19.723977 17372 raft_consensus.cc:686] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7a8704427b45a3b6327528df991b24, State: Running, Role: LEADER
I20260806 11:43:19.724294 17372 consensus_queue.cc:237] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } }
I20260806 11:43:19.726497 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.710574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:42873 (local address 127.0.113.3:44155)
0806 11:43:19.711847 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0806 11:43:19.711862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:43:19.711944 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:43:19.712208 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:43:19.712210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:43:19.712230 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:43:19.712381 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.712389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.713329 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.713332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.724647 (+ 11315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:43:19.724657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.725886 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.725890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:43:19.726060 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.726085 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 11:43:19.726163 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 11:43:19.726232 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":41,"thread_start_us":129,"threads_started":1}
I20260806 11:43:19.726944 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.711591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:44155 (local address 127.0.113.1:42873)
0806 11:43:19.711964 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:19.711968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:19.711983 (+ 15us) server_negotiation.cc:407] Connection header received
0806 11:43:19.712035 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:19.712038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:19.712084 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:19.712164 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:43:19.712520 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.713194 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.724782 (+ 11588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.725527 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.725661 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.725729 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:19.726783 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0806 11:43:19.726858 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260806 11:43:19.727530 16867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35faf3b8b2d4e1aaf339951d3a8650e" candidate_uuid: "5e7a8704427b45a3b6327528df991b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" is_pre_election: true
I20260806 11:43:19.727623 16866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35faf3b8b2d4e1aaf339951d3a8650e" candidate_uuid: "5e7a8704427b45a3b6327528df991b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d"
I20260806 11:43:19.727802 16867 raft_consensus.cc:2454] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7a8704427b45a3b6327528df991b24 in term 0.
I20260806 11:43:19.730834 16619 catalog_manager.cc:5495] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e7a8704427b45a3b6327528df991b24 (127.0.113.3). New cstate: current_term: 1 leader_uuid: "5e7a8704427b45a3b6327528df991b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7a8704427b45a3b6327528df991b24" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 } health_report { overall_health: UNKNOWN } } }
I20260806 11:43:19.778892 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.773426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.0.113.3:45741)
0806 11:43:19.773564 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 11:43:19.773569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:43:19.773584 (+ 15us) server_negotiation.cc:407] Connection header received
0806 11:43:19.773682 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:43:19.773685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:43:19.773764 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:43:19.773881 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:43:19.774437 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.775264 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.776595 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:43:19.776937 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:43:19.777039 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:43:19.777497 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:43:19.778061 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:43:19.778204 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 11:43:19.778659 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 11:43:19.778739 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 11:43:19.797919 16866 raft_consensus.cc:3037] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:43:19.798533 17050 raft_consensus.cc:1261] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [term 1 FOLLOWER]: Refusing update from remote peer 5e7a8704427b45a3b6327528df991b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:43:19.799307 17372 consensus_queue.cc:1035] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ce62e5e021428897b8c046a00f4a0e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:43:19.799559 16866 raft_consensus.cc:1261] T d35faf3b8b2d4e1aaf339951d3a8650e P 4ed5f31f5c924d0595b0ced91d9bc89d [term 1 FOLLOWER]: Refusing update from remote peer 5e7a8704427b45a3b6327528df991b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:43:19.803623 17372 consensus_queue.cc:1035] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed5f31f5c924d0595b0ced91d9bc89d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 42873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:43:19.815553 17395 mvcc.cc:205] Tried to move back new op lower bound from 7315523992764682240 to 7315523992485273600. Current Snapshot: MvccSnapshot[applied={T|T < 7315523992764682240}]
I20260806 11:43:19.819347 17393 mvcc.cc:205] Tried to move back new op lower bound from 7315523992764682240 to 7315523992485273600. Current Snapshot: MvccSnapshot[applied={T|T < 7315523992764682240}]
I20260806 11:43:19.829661 16619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:43:19.829856 16619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:43:19.832720 16619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c8556106bff47f1b77d28774d5b5c58: Sending DeleteTablet for 3 replicas of tablet d35faf3b8b2d4e1aaf339951d3a8650e
I20260806 11:43:19.833812 17221 tablet_service.cc:1514] Processing DeleteTablet for tablet d35faf3b8b2d4e1aaf339951d3a8650e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:43:19 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:36610
I20260806 11:43:19.835356 17041 tablet_service.cc:1514] Processing DeleteTablet for tablet d35faf3b8b2d4e1aaf339951d3a8650e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:43:19 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:37286
I20260806 11:43:19.836122 16834 tablet_service.cc:1514] Processing DeleteTablet for tablet d35faf3b8b2d4e1aaf339951d3a8650e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:43:19 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:52766
I20260806 11:43:19.837617 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 16740
I20260806 11:43:19.845275 17406 tablet_replica.cc:330] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: stopping tablet replica
I20260806 11:43:19.845642 17406 raft_consensus.cc:2229] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:43:19.846057 17406 raft_consensus.cc:2258] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:43:19.846580 17407 tablet_replica.cc:330] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: stopping tablet replica
I20260806 11:43:19.846967 17407 raft_consensus.cc:2229] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:43:19.847294 17407 raft_consensus.cc:2258] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:43:19.850025 17406 ts_tablet_manager.cc:1905] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:43:19.853672 17407 ts_tablet_manager.cc:1905] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:43:19.856361 17407 ts_tablet_manager.cc:1918] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:43:19.856467 17407 log.cc:1196] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/d35faf3b8b2d4e1aaf339951d3a8650e
I20260806 11:43:19.856814 17407 ts_tablet_manager.cc:1939] T d35faf3b8b2d4e1aaf339951d3a8650e P 44ce62e5e021428897b8c046a00f4a0e: Deleting consensus metadata
I20260806 11:43:19.861904 17406 ts_tablet_manager.cc:1918] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:43:19.862026 17406 log.cc:1196] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/d35faf3b8b2d4e1aaf339951d3a8650e
I20260806 11:43:19.862382 17406 ts_tablet_manager.cc:1939] T d35faf3b8b2d4e1aaf339951d3a8650e P 5e7a8704427b45a3b6327528df991b24: Deleting consensus metadata
I20260806 11:43:19.863766 16597 catalog_manager.cc:4841] TS 5e7a8704427b45a3b6327528df991b24 (127.0.113.3:45741): tablet d35faf3b8b2d4e1aaf339951d3a8650e (table test-table [id=86191ceba2a147af86a78b40563a1f73]) successfully deleted
W20260806 11:43:19.863862 16599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:42873 (error 108)
I20260806 11:43:19.864878 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 16946
I20260806 11:43:19.864889 16599 catalog_manager.cc:4841] TS 44ce62e5e021428897b8c046a00f4a0e (127.0.113.2:36283): tablet d35faf3b8b2d4e1aaf339951d3a8650e (table test-table [id=86191ceba2a147af86a78b40563a1f73]) successfully deleted
I20260806 11:43:19.883739 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:43:19.883376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:42873 (local address 127.0.0.1:52768)
0806 11:43:19.883570 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 11:43:19.883645 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260806 11:43:19.884011 16599 catalog_manager.cc:4584] TS 4ed5f31f5c924d0595b0ced91d9bc89d (127.0.113.1:42873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d35faf3b8b2d4e1aaf339951d3a8650e: Network error: Client connection negotiation failed: client connection to 127.0.113.1:42873: connect: Connection refused (error 111)
I20260806 11:43:19.900594 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 17145
I20260806 11:43:19.926471 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 16575
I20260806 11:43:19.939561 452 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 11:43:20.284574 452 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 11:43:20.284698 452 mini_ranger.cc:66] Stopping Ranger...
I20260806 11:43:20.629494 452 mini_ranger.cc:68] Stopped Ranger
2026-08-06 11:43:20.629 UTC [15974] LOG: received smart shutdown request
2026-08-06 11:43:20.634 UTC [15974] LOG: background worker "logical replication launcher" (PID 15988) exited with exit code 1
2026-08-06 11:43:20.634 UTC [15981] LOG: shutting down
2026-08-06 11:43:20.788 UTC [15974] LOG: database system is shut down
2026-08-06T11:43:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47300 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:43:20 dist-test-slave-45dx krb5kdc[17428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:43:20 dist-test-slave-45dx krb5kdc[17428](info): set up 2 sockets
Aug 06 11:43:20 dist-test-slave-45dx krb5kdc[17428](info): commencing operation
krb5kdc: starting...
W20260806 11:43:22.091457 452 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:43:22 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:43:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/rangeradmin_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/rangeradmin_127.0.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/rangerlookup_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/rangerlookup_127.0.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/HTTP_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/HTTP_127.0.113.20@KRBTEST.COM.keytab.
I20260806 11:43:22.269368 452 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskg6qW6_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/postgres -l logfile start
2026-08-06 11:43:24.607 UTC [17468] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 11:43:24.608 UTC [17468] LOG: listening on IPv4 address "127.0.113.20", port 40401
2026-08-06 11:43:24.611 UTC [17468] LOG: listening on Unix socket "/tmp/.s.PGSQL.40401"
2026-08-06 11:43:24.644 UTC [17473] LOG: database system was shut down at 2026-08-06 11:43:23 UTC
2026-08-06 11:43:24.648 UTC [17468] LOG: database system is ready to accept connections
I20260806 11:43:25.759599 452 mini_postgres.cc:96] Postgres bound to 40401
2026-08-06 11:43:25.767 UTC [17484] FATAL: database "slave" does not exist
127.0.113.20:40401 - accepting connections
I20260806 11:43:25.767563 452 mini_ranger.cc:161] Starting Ranger...
I20260806 11:43:25.782509 452 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 11:43:26.236577 452 mini_ranger.cc:87] Created ranger Postgres database
I20260806 11:43:26.236649 452 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-admin
2026-08-06 11:43:26,509 [I] DB FLAVOR :POSTGRES
2026-08-06 11:43:26,509 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 11:43:26,509 [I] Checking connection..
2026-08-06 11:43:26,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 11:43:26,741 [I] Checking connection passed.
2026-08-06 11:43:26,741 [I] --------- Verifying version history table ---------
2026-08-06 11:43:26,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 11:43:27,002 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 11:43:27,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 11:43:27,226 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 11:43:27,226 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 11:43:27,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 11:43:27.485 UTC [17575] WARNING: there is no transaction in progress
2026-08-06 11:43:27,495 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 11:43:27,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 11:43:27,750 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 11:43:27,751 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 11:43:27,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 11:43:28,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 11:43:28,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-45dx.c.gcp-upstream.internal','N') ;"
2026-08-06 11:43:28,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 11:43:28,790 [I] Table x_portal_user does not exist in database ranger
2026-08-06 11:43:28,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 11:43:29,036 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 11:43:29,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 11:43:29,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 11:43:29,572 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 11:43:29,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 11:43:30.530 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.547 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.563 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.578 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.594 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.611 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.665 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.672 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.687 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.697 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.707 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.720 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.730 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.736 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.742 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:30.749 UTC [17732] WARNING: there is no transaction in progress
2026-08-06 11:43:31,373 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 11:43:31,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 11:43:31,616 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 11:43:31,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 11:43:31,884 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 11:43:31,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 11:43:32,127 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 11:43:32,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 11:43:32,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 11:43:32,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-45dx.c.gcp-upstream.internal';"
2026-08-06 11:43:32,871 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 11:43:32,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 11:43:33,133 [I] DB_PATCHES have already been applied
I20260806 11:43:33.139365 452 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-admin:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskg6qW6_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 11:43:33.139430 452 mini_ranger.cc:193] Using host: 127.0.113.20
I20260806 11:43:33.143715 452 mini_ranger.cc:227] Ranger admin URL: http://127.0.113.20:43013
Aug 06, 2026 11:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 11:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 11:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 11:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 11:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 11:43:34.509449 452 mini_ranger.cc:160] Time spent starting Ranger: real 8.742s user 0.000s sys 0.011s
Aug 06, 2026 11:43:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 11:43:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 11:43:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 11:43:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 11:43:55 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 11:43:56.164019 452 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/rangerkms_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/rangerkms_127.0.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.113.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.113.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.113.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/HTTP_127.0.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.113.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/HTTP_127.0.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 11:43:56 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 11:43:56.248453 452 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 11:43:56.258126 452 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 11:43:56.997004 452 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 11:43:56.997098 452 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-kms
I20260806 11:43:56.997120 452 mini_ranger_kms.cc:227] Using postgres at 127.0.113.20:40401
2026-08-06 11:43:57,171 [I] DB FLAVOR :POSTGRES
2026-08-06 11:43:57,171 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 11:43:57,171 [I] Checking connection
2026-08-06 11:43:57,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 11:43:57,413 [I] connection success
2026-08-06 11:43:57,413 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 11:43:57,413 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 11:43:57,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 11:43:57,649 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 11:43:57,650 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 11:43:57,650 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 11:43:57,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.113.20:40401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 11:43:57,902 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 11:43:58.401386 452 mini_ranger_kms.cc:310] Created kms service
I20260806 11:43:58.594868 452 mini_ranger_kms.cc:326] Created kudu user
I20260806 11:43:58.639292 452 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 11:43:58 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:43:59.017787 452 mini_ranger_kms.cc:384] Added ranger policy
I20260806 11:43:59.017942 452 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-kms:/tmp/dist-test-taskg6qW6_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskg6qW6_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 11:43:59.017964 452 mini_ranger_kms.cc:240] Using host: 127.0.113.20
I20260806 11:43:59.022675 452 mini_ranger_kms.cc:276] Ranger KMS PID: 18001
I20260806 11:43:59.022768 452 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.113.20:60479
Aug 06, 2026 11:43:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:43:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:43:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 11:43:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 11:44:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 11:44:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 11:44:01 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786016641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.113.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:01.814416 452 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.566s user 0.010s sys 0.149s
I20260806 11:44:01.814554 452 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 11:44:01.814617 452 mini_ranger_kms.cc:401] 127.0.113.20:60479/kms/v1/keys
Aug 06 11:44:01 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.113.20@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
Aug 06 11:44:02 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.113.20@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
Aug 06 11:44:04 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:44:04.484972 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:38397
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:38397
--ranger_config_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:60479/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:04.602257 18085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:04.602525 18085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:04.602566 18085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:04.606472 18085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:04.606524 18085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:04.606556 18085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:04.606571 18085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:04.606586 18085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:04.612252 18085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.113.20:60479/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:38397
--ranger_config_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:04.613303 18085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:04.614337 18085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:04.625679 18090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:04.626217 18091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:04.626338 18095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:04.627765 18085 server_base.cc:1034] running on GCE node
I20260806 11:44:04.628564 18085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:04.630163 18085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:04.631356 18085 hybrid_clock.cc:648] HybridClock initialized: now 1786016644631331 us; error 41 us; skew 500 ppm
Aug 06 11:44:04 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:04.640973 18085 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:44:04.642203 18085 webserver.cc:460] Webserver started at http://127.0.113.62:36579/ using document root <none> and password file <none>
I20260806 11:44:04.642793 18085 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:04.642850 18085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:04.643100 18085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 11:44:04 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:04.691184 18085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "3f8de09a8b094222968bcc05cb608cde"
format_stamp: "Formatted at 2026-08-06 11:44:04 on dist-test-slave-45dx"
server_key: "47c0406ee145be578d82ad96c8513c71"
server_key_iv: "5506fdb92d1e2248163604d2e850da28"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:04.691876 18085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "3f8de09a8b094222968bcc05cb608cde"
format_stamp: "Formatted at 2026-08-06 11:44:04 on dist-test-slave-45dx"
server_key: "47c0406ee145be578d82ad96c8513c71"
server_key_iv: "5506fdb92d1e2248163604d2e850da28"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:04.695946 18085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 11:44:04.710104 18107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:04.711534 18085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 11:44:04.711687 18085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "3f8de09a8b094222968bcc05cb608cde"
format_stamp: "Formatted at 2026-08-06 11:44:04 on dist-test-slave-45dx"
server_key: "47c0406ee145be578d82ad96c8513c71"
server_key_iv: "5506fdb92d1e2248163604d2e850da28"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:04.711834 18085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:04.724118 18085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:04.727102 18085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:04.727309 18085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:04.735574 18085 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:38397
I20260806 11:44:04.735579 18183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:38397 every 8 connection(s)
I20260806 11:44:04.736603 18085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:44:04.739343 18184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:04.743304 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 18085
I20260806 11:44:04.743432 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:44:04.743746 452 external_mini_cluster.cc:1428] Setting key 6dea6a44cb6f947da7a887bce27b165b
I20260806 11:44:04.745436 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Bootstrap starting.
I20260806 11:44:04.747759 18184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:04.748741 18184 log.cc:824] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:04.750769 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: No bootstrap required, opened a new log
Aug 06 11:44:04 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:04.755153 18184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } }
I20260806 11:44:04.755460 18184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:04.755582 18184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8de09a8b094222968bcc05cb608cde, State: Initialized, Role: FOLLOWER
I20260806 11:44:04.756135 18184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } }
I20260806 11:44:04.756340 18184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:44:04.756412 18184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:44:04.756542 18184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:04.757431 18184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } }
I20260806 11:44:04.757841 18184 leader_election.cc:302] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f8de09a8b094222968bcc05cb608cde; no voters:
I20260806 11:44:04.758284 18184 leader_election.cc:288] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:44:04.758409 18190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:04.758674 18190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 1 LEADER]: Becoming Leader. State: Replica: 3f8de09a8b094222968bcc05cb608cde, State: Running, Role: LEADER
I20260806 11:44:04.759027 18190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } }
I20260806 11:44:04.759697 18184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:04.760358 18190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f8de09a8b094222968bcc05cb608cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } } }
I20260806 11:44:04.760560 18190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:04.760987 18190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f8de09a8b094222968bcc05cb608cde. Latest consensus state: current_term: 1 leader_uuid: "3f8de09a8b094222968bcc05cb608cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } } }
I20260806 11:44:04.761135 18190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:04.761449 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:04.745244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.0.113.62:38397)
0806 11:44:04.745748 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:04.745768 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:04.745813 (+ 45us) server_negotiation.cc:407] Connection header received
0806 11:44:04.746675 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:04.746697 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:04.747114 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:04.747479 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:04.748510 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:04.749980 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:04.751130 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:04.751811 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:04.751967 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:04.755141 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:04.755148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:04.755165 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:04.755203 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:04.757396 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:04.757967 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:04.757974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:04.757980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:04.758119 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:04.758507 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:04.758512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:04.758516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:04.759037 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:04.759212 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:04.759678 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 11:44:04.759985 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":148,"threads_started":1}
I20260806 11:44:04.762120 18184 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 11:44:04.761448 18204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:44:04.771931 18204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:44:04.786381 18204 catalog_manager.cc:1347] Generated new cluster ID: bc2368a9f35f43caa2baeb7bdebd1a96
I20260806 11:44:04.786571 18204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:44:04.809798 18204 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:44:04.811226 18204 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:44:04.815852 18204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Generated new TSK 0
I20260806 11:44:04.816843 18204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:44:05.539036 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:60479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38397
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:05.666421 18249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:05.666718 18249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:05.666764 18249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:05.671751 18249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:05.671823 18249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:05.671912 18249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:44:05.679715 18249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.113.20:60479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:05.681214 18249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:05.682646 18249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:05.697158 18265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:05.697162 18267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:05.698119 18264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:05.698760 18249 server_base.cc:1034] running on GCE node
I20260806 11:44:05.699421 18249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:05.700348 18249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:05.701565 18249 hybrid_clock.cc:648] HybridClock initialized: now 1786016645701528 us; error 44 us; skew 500 ppm
Aug 06 11:44:05 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:05.705303 18249 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:44:05.706425 18249 webserver.cc:460] Webserver started at http://127.0.113.1:44663/ using document root <none> and password file <none>
I20260806 11:44:05.706976 18249 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:05.707036 18249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:05.707319 18249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 11:44:05 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:05.724058 18249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "646a7227ca5c4fd49b9d984562b8be9d"
format_stamp: "Formatted at 2026-08-06 11:44:05 on dist-test-slave-45dx"
server_key: "5fa88bac82db55c703db3f78a1b6bb61"
server_key_iv: "b01a6bef1677ec622081c17b2ea14d95"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:05.725001 18249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "646a7227ca5c4fd49b9d984562b8be9d"
format_stamp: "Formatted at 2026-08-06 11:44:05 on dist-test-slave-45dx"
server_key: "5fa88bac82db55c703db3f78a1b6bb61"
server_key_iv: "b01a6bef1677ec622081c17b2ea14d95"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:05.733973 18249 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.004s
I20260806 11:44:05.770656 18280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:05.776700 18249 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260806 11:44:05.776885 18249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "646a7227ca5c4fd49b9d984562b8be9d"
format_stamp: "Formatted at 2026-08-06 11:44:05 on dist-test-slave-45dx"
server_key: "5fa88bac82db55c703db3f78a1b6bb61"
server_key_iv: "b01a6bef1677ec622081c17b2ea14d95"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:05.777037 18249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:05.793929 18249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:05.797844 18249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:05.798097 18249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:05.798744 18249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:05.799695 18249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:05.799744 18249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:05.799806 18249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:05.799844 18249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:05.811386 18249 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:36181
I20260806 11:44:05.811599 18451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:36181 every 8 connection(s)
I20260806 11:44:05.812856 18249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:44:05.822259 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 18249
I20260806 11:44:05.822393 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:44:05.822701 452 external_mini_cluster.cc:1428] Setting key 7582a186a8f17fed29f115528b9c914b
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
Aug 06 11:44:05 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:05.849865 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:05.824426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:49691 (local address 127.0.113.62:38397)
0806 11:44:05.828896 (+ 4470us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:05.828901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:05.833721 (+ 4820us) server_negotiation.cc:407] Connection header received
0806 11:44:05.835032 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:05.835037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:05.835103 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:05.835215 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:05.836882 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:05.837713 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:05.839855 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:05.840212 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:05.840321 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:05.845035 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:05.845040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:05.845043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:05.845078 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:05.847128 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:05.847828 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:05.847832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:05.847834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:05.847945 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:05.848416 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:05.848420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:05.848422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:05.848677 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:05.848761 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:05.849520 (+ 759us) server_negotiation.cc:299] Negotiation successful
0806 11:44:05.849664 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4367,"thread_start_us":86,"threads_started":1}
I20260806 11:44:05.851961 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:05.828502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:38397 (local address 127.0.113.1:49691)
0806 11:44:05.833517 (+ 5015us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:05.833594 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:05.834737 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:05.835561 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:05.835571 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:05.835946 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:05.836693 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:05.836713 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:05.837884 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:05.837888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:05.839303 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:05.839315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:05.840576 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:05.840580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:05.840799 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:05.841728 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:05.841751 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:05.844794 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:05.847292 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:05.847300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:05.847319 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:05.847698 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:05.848069 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:05.848072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:05.848074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:05.848266 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:05.848840 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:05.848847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:05.849203 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 11:44:05.850379 (+ 1176us) client_negotiation.cc:240] Negotiation successful
0806 11:44:05.850774 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4730,"thread_start_us":168,"threads_started":1}
I20260806 11:44:05.854120 18452 heartbeater.cc:347] Connected to a master server at 127.0.113.62:38397
I20260806 11:44:05.854408 18452 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:05.855064 18452 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
I20260806 11:44:05.857024 18131 ts_manager.cc:194] Registered new tserver with Master: 646a7227ca5c4fd49b9d984562b8be9d (127.0.113.1:36181)
I20260806 11:44:05.858626 18131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:49691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:44:05.868659 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:60479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38397
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:44:05 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786016645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 11:44:06.035135 18464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:06.035640 18464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:06.035769 18464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:06.042079 18464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:06.042181 18464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:06.042271 18464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:44:06.050387 18464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.113.20:60479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:06.051841 18464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:06.053253 18464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:06.065945 18477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:06.065919 18481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:06.066453 18478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:06.067423 18464 server_base.cc:1034] running on GCE node
I20260806 11:44:06.067941 18464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:06.068876 18464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:06.070096 18464 hybrid_clock.cc:648] HybridClock initialized: now 1786016646070069 us; error 35 us; skew 500 ppm
Aug 06 11:44:06 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:06.074127 18464 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:44:06.075253 18464 webserver.cc:460] Webserver started at http://127.0.113.2:44031/ using document root <none> and password file <none>
I20260806 11:44:06.075991 18464 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:06.076087 18464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:06.076372 18464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 11:44:06 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:06.097580 18464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "80a98de7a10e496ea43e0aa542a1b02d"
format_stamp: "Formatted at 2026-08-06 11:44:06 on dist-test-slave-45dx"
server_key: "e800781800e9f24d4871b6c1c0290577"
server_key_iv: "b250e570352a97f778e77928a00b609a"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:06.098856 18464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "80a98de7a10e496ea43e0aa542a1b02d"
format_stamp: "Formatted at 2026-08-06 11:44:06 on dist-test-slave-45dx"
server_key: "e800781800e9f24d4871b6c1c0290577"
server_key_iv: "b250e570352a97f778e77928a00b609a"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:06.105335 18464 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 11:44:06.126399 18490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:06.129802 18464 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260806 11:44:06.130035 18464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "80a98de7a10e496ea43e0aa542a1b02d"
format_stamp: "Formatted at 2026-08-06 11:44:06 on dist-test-slave-45dx"
server_key: "e800781800e9f24d4871b6c1c0290577"
server_key_iv: "b250e570352a97f778e77928a00b609a"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:06.130266 18464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:06.167649 18464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:06.172168 18464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:06.172508 18464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:06.173457 18464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:06.174738 18464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:06.174800 18464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:06.174852 18464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:06.174937 18464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:06.200668 18464 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:37969
I20260806 11:44:06.201064 18681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:37969 every 8 connection(s)
I20260806 11:44:06.202066 18464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:44:06.210898 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 18464
I20260806 11:44:06.211082 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:44:06.211452 452 external_mini_cluster.cc:1428] Setting key c22a52322ac3d867625b9cebea032f5d
Aug 06 11:44:06 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:06.222131 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:06.206684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:34191 (local address 127.0.113.62:38397)
0806 11:44:06.206827 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:06.206832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:06.207322 (+ 490us) server_negotiation.cc:407] Connection header received
0806 11:44:06.208681 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:06.208685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:06.208747 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:06.208823 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:06.210612 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:06.211490 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:06.213989 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:06.214471 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:06.214618 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:06.217882 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:06.217886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:06.217888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:06.217917 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:06.219606 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:06.220333 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:06.220337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:06.220339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:06.220430 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:06.220832 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:06.220835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:06.220837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:06.221042 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:06.221161 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:06.221756 (+ 595us) server_negotiation.cc:299] Negotiation successful
0806 11:44:06.221917 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:44:06.223168 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:06.205519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:38397 (local address 127.0.113.2:34191)
0806 11:44:06.207151 (+ 1632us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:06.207210 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:06.208403 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:06.209276 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:06.209286 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:06.209683 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:06.210422 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:06.210445 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:06.211777 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:06.211781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:06.213834 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:06.213846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:06.214675 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:06.214680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:06.214930 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:06.215740 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:06.215763 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:06.217664 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:06.219748 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:06.219755 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:06.219783 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:06.220181 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:06.220544 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:06.220547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:06.220549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:06.220717 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:06.221175 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:06.221184 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:06.221492 (+ 308us) client_negotiation.cc:769] Sending connection context
0806 11:44:06.221697 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 11:44:06.222079 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"thread_start_us":157,"threads_started":1}
I20260806 11:44:06.224622 18682 heartbeater.cc:347] Connected to a master server at 127.0.113.62:38397
I20260806 11:44:06.224915 18682 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:06.225543 18682 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
I20260806 11:44:06.226797 18131 ts_manager.cc:194] Registered new tserver with Master: 80a98de7a10e496ea43e0aa542a1b02d (127.0.113.2:37969)
I20260806 11:44:06.227417 18131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:34191
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:44:06.253556 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:60479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38397
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:44:06 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
W20260806 11:44:06.421727 18693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:06.422223 18693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:06.422338 18693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:06.428193 18693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:06.428375 18693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:06.428504 18693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:44:06.435890 18693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.113.20:60479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:06.437477 18693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:06.438855 18693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:06.452227 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:06.453205 18706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:06.452631 18693 server_base.cc:1034] running on GCE node
W20260806 11:44:06.452471 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:06.455945 18693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:06.456682 18693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:06.457858 18693 hybrid_clock.cc:648] HybridClock initialized: now 1786016646457797 us; error 76 us; skew 500 ppm
Aug 06 11:44:06 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:06.461494 18693 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:44:06.462525 18693 webserver.cc:460] Webserver started at http://127.0.113.3:43939/ using document root <none> and password file <none>
I20260806 11:44:06.463179 18693 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:06.463249 18693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:06.463474 18693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 11:44:06 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:06.478444 18693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "a16624c04cf042b3be5e1817ba04043c"
format_stamp: "Formatted at 2026-08-06 11:44:06 on dist-test-slave-45dx"
server_key: "4b88a69a2ed38188933064db331dd28b"
server_key_iv: "11848d07a001749185096cc1d57d7ba6"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:06.479214 18693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "a16624c04cf042b3be5e1817ba04043c"
format_stamp: "Formatted at 2026-08-06 11:44:06 on dist-test-slave-45dx"
server_key: "4b88a69a2ed38188933064db331dd28b"
server_key_iv: "11848d07a001749185096cc1d57d7ba6"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:06.484699 18693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 11:44:06.500114 18715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:06.501732 18693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:44:06.501910 18693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "a16624c04cf042b3be5e1817ba04043c"
format_stamp: "Formatted at 2026-08-06 11:44:06 on dist-test-slave-45dx"
server_key: "4b88a69a2ed38188933064db331dd28b"
server_key_iv: "11848d07a001749185096cc1d57d7ba6"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:06.502084 18693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:06.520730 18693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:06.524013 18693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:06.524231 18693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:06.524963 18693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:06.525883 18693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:06.525954 18693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:06.526024 18693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:06.526074 18693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:06.544718 18693 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:33105
I20260806 11:44:06.544904 18909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:33105 every 8 connection(s)
I20260806 11:44:06.545925 18693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:44:06.554994 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 18693
I20260806 11:44:06.555124 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:44:06.555440 452 external_mini_cluster.cc:1428] Setting key 61a28cb004f9aba2b91a4ef11937f8a1
Aug 06 11:44:06 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:06.563534 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:06.548555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:33113 (local address 127.0.113.62:38397)
0806 11:44:06.548712 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:06.548717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:06.549678 (+ 961us) server_negotiation.cc:407] Connection header received
0806 11:44:06.550999 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:06.551003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:06.551060 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:06.551111 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:06.552896 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:06.553708 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:06.555383 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:06.555692 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:06.555761 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:06.559013 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:06.559017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:06.559020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:06.559050 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:06.560878 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:06.561640 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:06.561644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:06.561646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:06.561733 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:06.562199 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:06.562203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:06.562205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:06.562450 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:06.562561 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:06.563224 (+ 663us) server_negotiation.cc:299] Negotiation successful
0806 11:44:06.563358 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:44:06.564507 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:06.548787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:38397 (local address 127.0.113.3:33113)
0806 11:44:06.549499 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:06.549563 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:06.550742 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:06.551555 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:06.551564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:06.551935 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:06.552665 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:06.552683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:06.553868 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:06.553872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:06.555233 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:06.555244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:06.555973 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:06.555977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:06.556177 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:06.556981 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:06.557002 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:06.558803 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:06.561043 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:06.561049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:06.561066 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:06.561512 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:06.561875 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:06.561879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:06.561881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:06.562069 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:06.562579 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:06.562585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:06.562917 (+ 332us) client_negotiation.cc:769] Sending connection context
0806 11:44:06.563130 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 11:44:06.563489 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":192,"threads_started":1}
I20260806 11:44:06.565904 18912 heartbeater.cc:347] Connected to a master server at 127.0.113.62:38397
I20260806 11:44:06.566167 18912 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:06.566752 18912 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
I20260806 11:44:06.567917 18131 ts_manager.cc:194] Registered new tserver with Master: a16624c04cf042b3be5e1817ba04043c (127.0.113.3:33105)
I20260806 11:44:06.568657 18131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:33113
I20260806 11:44:06.570482 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:44:06.571332 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 18249
I20260806 11:44:06.606712 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 18464
I20260806 11:44:06.639926 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 18693
I20260806 11:44:06.675120 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 18085
I20260806 11:44:06.688546 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:38397
--webserver_interface=127.0.113.62
--webserver_port=36579
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:38397
--ranger_config_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:60479/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.113.20:60479/kms with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:06.799032 18919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:06.799275 18919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:06.799320 18919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:06.803126 18919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:06.803175 18919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:06.803194 18919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:06.803210 18919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:06.803224 18919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:06.808269 18919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.113.20:60479/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:38397
--ranger_config_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=36579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:06.809201 18919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:06.810124 18919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:06.818455 18926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:06.818483 18929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:06.818643 18927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:06.818679 18919 server_base.cc:1034] running on GCE node
I20260806 11:44:06.819372 18919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:06.820465 18919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:06.821651 18919 hybrid_clock.cc:648] HybridClock initialized: now 1786016646821618 us; error 43 us; skew 500 ppm
Aug 06 11:44:06 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:06.824525 18919 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:44:06.825443 18919 webserver.cc:460] Webserver started at http://127.0.113.62:36579/ using document root <none> and password file <none>
I20260806 11:44:06.825965 18919 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:06.826066 18919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 11:44:06 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:06.875931 18919 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 11:44:06.878320 18945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:06.879769 18919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 11:44:06.879925 18919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "3f8de09a8b094222968bcc05cb608cde"
format_stamp: "Formatted at 2026-08-06 11:44:04 on dist-test-slave-45dx"
server_key: "47c0406ee145be578d82ad96c8513c71"
server_key_iv: "5506fdb92d1e2248163604d2e850da28"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:06.880589 18919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:06.899501 18919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:06.902290 18919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:06.902498 18919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:06.909770 18919 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:38397
I20260806 11:44:06.909778 19042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:38397 every 8 connection(s)
I20260806 11:44:06.910806 18919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:44:06.914968 19043 sys_catalog.cc:263] Verifying existing consensus state
I20260806 11:44:06.916198 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 18919
I20260806 11:44:06.916633 19043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Bootstrap starting.
I20260806 11:44:06.916646 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:60479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:36181
--local_ip_for_outbound_sockets=127.0.113.1
--tserver_master_addrs=127.0.113.62:38397
--webserver_port=44663
--webserver_interface=127.0.113.1
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.113.20:60479/kms with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:44:06.933146 19043 log.cc:824] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:06.939922 19043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:44:06.940428 19043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Bootstrap complete.
I20260806 11:44:06.944641 19043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } }
I20260806 11:44:06.945068 19043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8de09a8b094222968bcc05cb608cde, State: Initialized, Role: FOLLOWER
I20260806 11:44:06.945595 19043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } }
I20260806 11:44:06.945739 19043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:44:06.945827 19043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:44:06.945945 19043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 1 FOLLOWER]: Advancing to term 2
I20260806 11:44:06.947850 19043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } }
I20260806 11:44:06.948194 19043 leader_election.cc:302] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f8de09a8b094222968bcc05cb608cde; no voters:
I20260806 11:44:06.948586 19043 leader_election.cc:288] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 11:44:06.948690 19054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 2 FOLLOWER]: Leader election won for term 2
I20260806 11:44:06.948910 19054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [term 2 LEADER]: Becoming Leader. State: Replica: 3f8de09a8b094222968bcc05cb608cde, State: Running, Role: LEADER
I20260806 11:44:06.949270 19054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } }
I20260806 11:44:06.949864 19043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:06.950832 19056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f8de09a8b094222968bcc05cb608cde. Latest consensus state: current_term: 2 leader_uuid: "3f8de09a8b094222968bcc05cb608cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } } }
I20260806 11:44:06.950968 19056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:06.951282 19055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f8de09a8b094222968bcc05cb608cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8de09a8b094222968bcc05cb608cde" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38397 } } }
I20260806 11:44:06.951364 19055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:06.951785 19067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:44:06.952265 19043 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 11:44:06.958925 19067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:44:06.959851 19067 catalog_manager.cc:1259] Loaded cluster ID: bc2368a9f35f43caa2baeb7bdebd1a96
I20260806 11:44:06.959975 19067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:44:06.962903 19067 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:44:06.963793 19067 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Loaded TSK: 0
I20260806 11:44:06.965265 19067 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 11:44:07.053974 19046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:07.054302 19046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:07.054374 19046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:07.059736 19046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:07.059830 19046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:07.059932 19046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:44:07.066000 19046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.113.20:60479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:36181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=44663
--tserver_master_addrs=127.0.113.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:07.067216 19046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:07.068499 19046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:07.081306 19104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:07.081543 19109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:07.081321 19106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:07.081833 19046 server_base.cc:1034] running on GCE node
I20260806 11:44:07.082449 19046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:07.083088 19046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:07.084275 19046 hybrid_clock.cc:648] HybridClock initialized: now 1786016647084248 us; error 43 us; skew 500 ppm
Aug 06 11:44:07 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:07.087738 19046 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:44:07.088799 19046 webserver.cc:460] Webserver started at http://127.0.113.1:44663/ using document root <none> and password file <none>
I20260806 11:44:07.089375 19046 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:07.089483 19046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 11:44:07 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:07.112920 19046 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260806 11:44:07.115448 19123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:07.117059 19046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:44:07.117268 19046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "646a7227ca5c4fd49b9d984562b8be9d"
format_stamp: "Formatted at 2026-08-06 11:44:05 on dist-test-slave-45dx"
server_key: "5fa88bac82db55c703db3f78a1b6bb61"
server_key_iv: "b01a6bef1677ec622081c17b2ea14d95"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:07.118067 19046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:07.140012 19046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:07.143858 19046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:07.144120 19046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:07.144888 19046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:07.145982 19046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:07.146093 19046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:07.146236 19046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:07.146340 19046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:07.165563 19046 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:36181
I20260806 11:44:07.166981 19046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:44:07.169658 19296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:36181 every 8 connection(s)
I20260806 11:44:07.176748 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 19046
I20260806 11:44:07.177322 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:60479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:37969
--local_ip_for_outbound_sockets=127.0.113.2
--tserver_master_addrs=127.0.113.62:38397
--webserver_port=44031
--webserver_interface=127.0.113.2
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.113.20:60479/kms with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:44:07 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:07.191425 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:07.172674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:54033 (local address 127.0.113.62:38397)
0806 11:44:07.173877 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:07.173899 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:07.174368 (+ 469us) server_negotiation.cc:407] Connection header received
0806 11:44:07.175676 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:07.175693 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:07.176083 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:07.177198 (+ 1115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:07.177897 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.179409 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.181136 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.181880 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.182021 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:07.185404 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:07.185414 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:07.185433 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:07.185471 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:07.187609 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:07.188219 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:07.188225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:07.188230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:07.188408 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:07.188798 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:07.188801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:07.188803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:07.189257 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:07.189386 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:07.190091 (+ 705us) server_negotiation.cc:299] Negotiation successful
0806 11:44:07.190413 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":116,"threads_started":1}
I20260806 11:44:07.193024 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:07.172806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:38397 (local address 127.0.113.1:54033)
0806 11:44:07.174179 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:07.174231 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:07.175213 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:07.176543 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:07.176551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:07.177008 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:07.177676 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:07.177695 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.179594 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.179597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:07.180986 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:07.180998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.182286 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.182290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:07.182490 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:07.183256 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:07.183281 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:07.185122 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:07.187747 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:07.187753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:07.187765 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:07.188081 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:07.188527 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:07.188530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:07.188532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:07.188690 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:07.189455 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:07.189461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:07.189747 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 11:44:07.191839 (+ 2092us) client_negotiation.cc:240] Negotiation successful
0806 11:44:07.192130 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"thread_start_us":132,"threads_started":1}
I20260806 11:44:07.196188 19297 heartbeater.cc:347] Connected to a master server at 127.0.113.62:38397
I20260806 11:44:07.196494 19297 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:07.197124 19297 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
W20260806 11:44:07.199754 19297 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:38397 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 11:44:07.199894 19297 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:07.200100 19297 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
I20260806 11:44:07.201115 19297 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:07.201287 19297 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
W20260806 11:44:07.202144 19297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 11:44:07.322837 19301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:07.323289 19301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:07.323408 19301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:07.327746 19301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:07.327836 19301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:07.327935 19301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:44:07.333596 19301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.113.20:60479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:37969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=44031
--tserver_master_addrs=127.0.113.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:07.334749 19301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:07.335821 19301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:07.345851 19309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:07.346153 19313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:07.346490 19301 server_base.cc:1034] running on GCE node
W20260806 11:44:07.346524 19310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:07.347074 19301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:07.347877 19301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:07.349114 19301 hybrid_clock.cc:648] HybridClock initialized: now 1786016647349075 us; error 49 us; skew 500 ppm
Aug 06 11:44:07 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:07.352728 19301 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:44:07.353578 19301 webserver.cc:460] Webserver started at http://127.0.113.2:44031/ using document root <none> and password file <none>
I20260806 11:44:07.354143 19301 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:07.354218 19301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 11:44:07 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:07.393298 19301 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.000s
I20260806 11:44:07.403733 19327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:07.405524 19301 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260806 11:44:07.405663 19301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "80a98de7a10e496ea43e0aa542a1b02d"
format_stamp: "Formatted at 2026-08-06 11:44:06 on dist-test-slave-45dx"
server_key: "e800781800e9f24d4871b6c1c0290577"
server_key_iv: "b250e570352a97f778e77928a00b609a"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:07.406420 19301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:07.434172 19301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:07.438274 19301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:07.438563 19301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:07.439451 19301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:07.440595 19301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:07.440704 19301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:07.440815 19301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:07.440882 19301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:07.455521 19301 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:37969
I20260806 11:44:07.455581 19506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:37969 every 8 connection(s)
I20260806 11:44:07.457036 19301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:44:07.460937 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 19301
I20260806 11:44:07.461455 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.113.20:60479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:33105
--local_ip_for_outbound_sockets=127.0.113.3
--tserver_master_addrs=127.0.113.62:38397
--webserver_port=43939
--webserver_interface=127.0.113.3
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.113.20:60479/kms with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:44:07 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:07.476526 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:07.463807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:43237 (local address 127.0.113.62:38397)
0806 11:44:07.463948 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:07.463953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:07.463970 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:44:07.464023 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:07.464027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:07.464076 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:07.464164 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:07.465811 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.466623 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.468405 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.468739 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.468840 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:07.472102 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:07.472107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:07.472110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:07.472139 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:07.473950 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:07.474583 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:07.474586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:07.474589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:07.474676 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:07.475114 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:07.475118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:07.475119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:07.475354 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:07.475463 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:07.476184 (+ 721us) server_negotiation.cc:299] Negotiation successful
0806 11:44:07.476354 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 11:44:07.478363 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:07.459701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:38397 (local address 127.0.113.2:43237)
0806 11:44:07.460272 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:07.460363 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:07.461370 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:07.464534 (+ 3164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:07.464544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:07.464911 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:07.465635 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:07.465659 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.466825 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.466828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:07.468220 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:07.468233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.469095 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.469101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:07.469308 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:07.470060 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:07.470084 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:07.471900 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:07.474093 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:07.474099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:07.474114 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:07.474456 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:07.474788 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:07.474791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:07.474793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:07.474972 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:07.475538 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:07.475544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:07.475887 (+ 343us) client_negotiation.cc:769] Sending connection context
0806 11:44:07.476996 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0806 11:44:07.477335 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":137,"threads_started":1}
I20260806 11:44:07.479955 19507 heartbeater.cc:347] Connected to a master server at 127.0.113.62:38397
I20260806 11:44:07.480194 19507 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:07.480918 19507 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
W20260806 11:44:07.482692 19507 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:38397 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 11:44:07.482826 19507 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:07.482980 19507 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
I20260806 11:44:07.484002 19507 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:07.484170 19507 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
W20260806 11:44:07.485085 19507 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 11:44:07.623719 19513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:07.624034 19513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:07.624092 19513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:07.629776 19513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:07.629915 19513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:07.630043 19513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:44:07.637213 19513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.113.20:60479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:33105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=43939
--tserver_master_addrs=127.0.113.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:07.638760 19513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:07.640079 19513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:07.652680 19523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:07.652704 19519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:07.653909 19513 server_base.cc:1034] running on GCE node
W20260806 11:44:07.656757 19525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:07.657259 19513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:07.657974 19513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:07.659188 19513 hybrid_clock.cc:648] HybridClock initialized: now 1786016647659107 us; error 81 us; skew 500 ppm
Aug 06 11:44:07 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:07.662920 19513 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:44:07.663959 19513 webserver.cc:460] Webserver started at http://127.0.113.3:43939/ using document root <none> and password file <none>
I20260806 11:44:07.664665 19513 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:07.664809 19513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 11:44:07 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:07.684224 19513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 11:44:07.686818 19536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:07.688295 19513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 11:44:07.688444 19513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "a16624c04cf042b3be5e1817ba04043c"
format_stamp: "Formatted at 2026-08-06 11:44:06 on dist-test-slave-45dx"
server_key: "4b88a69a2ed38188933064db331dd28b"
server_key_iv: "11848d07a001749185096cc1d57d7ba6"
server_key_version: "kuduclusterkey@0"
I20260806 11:44:07.689188 19513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:07.719080 19513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:07.723652 19513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:07.723912 19513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:07.724651 19513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:07.725682 19513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:07.725775 19513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:07.725857 19513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:07.725906 19513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:07.735939 19513 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:33105
I20260806 11:44:07.737222 19513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:44:07.742069 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 19513
I20260806 11:44:07.757400 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:07.742865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.0.113.62:38397)
0806 11:44:07.743064 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:07.743071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:07.743238 (+ 167us) server_negotiation.cc:407] Connection header received
0806 11:44:07.743392 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:07.743397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:07.743460 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:07.743565 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:07.745548 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.746430 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.747998 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.748439 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.748545 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:07.753657 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:07.753661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:07.753664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:07.753700 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:07.755501 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:07.755995 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:07.755999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:07.756001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:07.756099 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:07.756423 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:07.756426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:07.756428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:07.756666 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:07.756917 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:07.757099 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 11:44:07.757230 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 11:44:07.757687 19708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:33105 every 8 connection(s)
Aug 06 11:44:07 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:07.774457 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:07.760043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:38397 (local address 127.0.113.3:32937)
0806 11:44:07.760567 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:07.760622 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:07.761597 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:07.762224 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:07.762233 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:07.762576 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:07.763199 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:07.763215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.764274 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.764278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:07.765583 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:07.765595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.766266 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.766270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:07.766470 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:07.767113 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:07.767133 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:07.768842 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:07.771385 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:07.771390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:07.771404 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:07.771721 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:07.772009 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:07.772012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:07.772014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:07.772176 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:07.772648 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:07.772654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:07.772966 (+ 312us) client_negotiation.cc:769] Sending connection context
0806 11:44:07.773150 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 11:44:07.773467 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":164,"threads_started":1}
I20260806 11:44:07.775206 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:07.759573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:32937 (local address 127.0.113.62:38397)
0806 11:44:07.759716 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:07.759723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:07.760731 (+ 1008us) server_negotiation.cc:407] Connection header received
0806 11:44:07.761884 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:07.761888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:07.761937 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:07.762023 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:07.763354 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.764129 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.765718 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:07.766023 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:07.766112 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:07.769022 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:07.769026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:07.769028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:07.769053 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:07.771237 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:07.771833 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:07.771836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:07.771838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:07.771917 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:07.772274 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:07.772276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:07.772278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:07.772505 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:07.772592 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:07.774951 (+ 2359us) server_negotiation.cc:299] Negotiation successful
0806 11:44:07.775062 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 11:44:07.779062 19709 heartbeater.cc:347] Connected to a master server at 127.0.113.62:38397
I20260806 11:44:07.779345 19709 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:07.779969 19709 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
I20260806 11:44:07.782387 18983 ts_manager.cc:194] Registered new tserver with Master: a16624c04cf042b3be5e1817ba04043c (127.0.113.3:33105)
I20260806 11:44:07.784035 18983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:32937
Aug 06 11:44:08 dist-test-slave-45dx krb5kdc[17428](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786016648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:08.202384 19297 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:08.202706 19297 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
I20260806 11:44:08.203611 18983 ts_manager.cc:194] Registered new tserver with Master: 646a7227ca5c4fd49b9d984562b8be9d (127.0.113.1:36181)
I20260806 11:44:08.204442 18983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:54033
I20260806 11:44:08.485324 19507 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:08.485630 19507 heartbeater.cc:511] Master 127.0.113.62:38397 requested a full tablet report, sending...
I20260806 11:44:08.486426 18983 ts_manager.cc:194] Registered new tserver with Master: 80a98de7a10e496ea43e0aa542a1b02d (127.0.113.2:37969)
I20260806 11:44:08.487172 18983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:43237
I20260806 11:44:08.495589 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:44:08.510154 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.497204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.0.113.62:38397)
0806 11:44:08.497469 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:08.497473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:08.497595 (+ 122us) server_negotiation.cc:407] Connection header received
0806 11:44:08.497728 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:08.497731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:08.497774 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:08.497864 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:08.498853 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.499711 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.501706 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.501975 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.502085 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.505919 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:08.505922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:08.505925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:08.505950 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:08.507923 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:08.508463 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:08.508467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:08.508469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:08.508560 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:08.508859 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:08.508862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:08.508864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:08.509077 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:08.509202 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:08.509861 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 11:44:08.509988 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":117,"threads_started":1}
I20260806 11:44:08.514403 18983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:44:08.518317 18983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 06 11:44:08 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:08.538769 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.530125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:37969 (local address 127.0.0.1:41164)
0806 11:44:08.533123 (+ 2998us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:08.533169 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:08.533308 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:08.534039 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:08.534045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:08.534069 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:08.534187 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.534193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.535572 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.535575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.536982 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.536993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.538274 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.538278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.538381 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.538416 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 11:44:08.538511 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 11:44:08.538618 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2813,"thread_start_us":152,"threads_started":1}
I20260806 11:44:08.539182 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.532613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.0.113.1:36181)
0806 11:44:08.532886 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:08.532893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:08.533758 (+ 865us) server_negotiation.cc:407] Connection header received
0806 11:44:08.533841 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:08.533846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:08.534045 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:08.534182 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:08.534659 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.535805 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.537273 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.538089 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.538196 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.538328 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:08.538936 (+ 608us) server_negotiation.cc:299] Negotiation successful
0806 11:44:08.539039 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260806 11:44:08.539192 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.531158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.0.113.2:37969)
0806 11:44:08.531544 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:08.531555 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:08.533282 (+ 1727us) server_negotiation.cc:407] Connection header received
0806 11:44:08.533466 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:08.533473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:08.533668 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:08.533838 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:08.534311 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.535432 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.537117 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.537963 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.538042 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.538179 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:08.538975 (+ 796us) server_negotiation.cc:299] Negotiation successful
0806 11:44:08.539069 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":115,"threads_started":1}
I20260806 11:44:08.538832 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.532083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:36181 (local address 127.0.0.1:57454)
0806 11:44:08.533540 (+ 1457us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:08.533553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:08.533650 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:08.534406 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:08.534411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:08.534432 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:08.534527 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.534534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.535947 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.535951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.537118 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.537130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.538407 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.538410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.538502 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.538526 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 11:44:08.538608 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 11:44:08.538682 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"thread_start_us":90,"threads_started":1}
I20260806 11:44:08.542626 19408 tablet_service.cc:1467] Processing CreateTablet for tablet b529f868e888470a82888d40b92ea651 (DEFAULT_TABLE table=test-table [id=be4e4b6380034cdd9a79e53371701ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:08.543773 19408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b529f868e888470a82888d40b92ea651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:08.543908 19178 tablet_service.cc:1467] Processing CreateTablet for tablet b529f868e888470a82888d40b92ea651 (DEFAULT_TABLE table=test-table [id=be4e4b6380034cdd9a79e53371701ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:08.544899 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b529f868e888470a82888d40b92ea651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:08.546705 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.532279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:33105 (local address 127.0.0.1:33200)
0806 11:44:08.539414 (+ 7135us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:08.539428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:08.539496 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:08.542496 (+ 3000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:08.542499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:08.542516 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:08.542585 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.542590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.543989 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.543992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.545076 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.545087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.546302 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.546305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.546400 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.546426 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:44:08.546506 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 11:44:08.546571 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7085,"thread_start_us":102,"threads_started":1}
I20260806 11:44:08.547206 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.541595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.0.113.3:33105)
0806 11:44:08.541893 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:08.541901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:08.541928 (+ 27us) server_negotiation.cc:407] Connection header received
0806 11:44:08.542004 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:08.542008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:08.542190 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:08.542336 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:08.542721 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.543846 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.545218 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.545962 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.546081 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.546216 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:08.546969 (+ 753us) server_negotiation.cc:299] Negotiation successful
0806 11:44:08.547074 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":112,"threads_started":1}
I20260806 11:44:08.549870 19604 tablet_service.cc:1467] Processing CreateTablet for tablet b529f868e888470a82888d40b92ea651 (DEFAULT_TABLE table=test-table [id=be4e4b6380034cdd9a79e53371701ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:08.550887 19604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b529f868e888470a82888d40b92ea651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:08.556689 19755 tablet_bootstrap.cc:492] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d: Bootstrap starting.
I20260806 11:44:08.557161 19754 tablet_bootstrap.cc:492] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: Bootstrap starting.
I20260806 11:44:08.559846 19754 tablet_bootstrap.cc:654] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:08.562228 19755 tablet_bootstrap.cc:654] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:08.563198 19755 log.cc:824] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:08.564411 19754 log.cc:824] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:08.567416 19755 tablet_bootstrap.cc:492] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d: No bootstrap required, opened a new log
I20260806 11:44:08.567724 19755 ts_tablet_manager.cc:1397] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260806 11:44:08.568866 19756 tablet_bootstrap.cc:492] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: Bootstrap starting.
I20260806 11:44:08.571337 19756 tablet_bootstrap.cc:654] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:08.571959 19755 raft_consensus.cc:348] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.572177 19756 log.cc:824] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:08.572366 19755 raft_consensus.cc:374] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:08.572496 19755 raft_consensus.cc:729] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646a7227ca5c4fd49b9d984562b8be9d, State: Initialized, Role: FOLLOWER
I20260806 11:44:08.573073 19755 consensus_queue.cc:260] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.573868 19297 heartbeater.cc:503] Master 127.0.113.62:38397 was elected leader, sending a full tablet report...
I20260806 11:44:08.574231 19755 ts_tablet_manager.cc:1428] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260806 11:44:08.578925 19754 tablet_bootstrap.cc:492] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: No bootstrap required, opened a new log
I20260806 11:44:08.579224 19754 ts_tablet_manager.cc:1397] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20260806 11:44:08.583236 19754 raft_consensus.cc:348] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.583611 19754 raft_consensus.cc:374] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:08.583700 19754 raft_consensus.cc:729] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a98de7a10e496ea43e0aa542a1b02d, State: Initialized, Role: FOLLOWER
I20260806 11:44:08.584271 19754 consensus_queue.cc:260] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.585482 19754 ts_tablet_manager.cc:1428] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 11:44:08.585898 19507 heartbeater.cc:503] Master 127.0.113.62:38397 was elected leader, sending a full tablet report...
I20260806 11:44:08.586346 19756 tablet_bootstrap.cc:492] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: No bootstrap required, opened a new log
I20260806 11:44:08.586544 19756 ts_tablet_manager.cc:1397] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260806 11:44:08.590952 19756 raft_consensus.cc:348] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.591297 19756 raft_consensus.cc:374] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:08.591403 19756 raft_consensus.cc:729] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16624c04cf042b3be5e1817ba04043c, State: Initialized, Role: FOLLOWER
I20260806 11:44:08.591898 19756 consensus_queue.cc:260] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.593075 19756 ts_tablet_manager.cc:1428] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 11:44:08.593449 19709 heartbeater.cc:503] Master 127.0.113.62:38397 was elected leader, sending a full tablet report...
W20260806 11:44:08.670790 19298 tablet.cc:2369] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:44:08.709240 19509 tablet.cc:2369] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:44:08.762137 19710 tablet.cc:2369] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:44:08.765936 19763 raft_consensus.cc:482] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:08.766175 19763 raft_consensus.cc:504] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.766557 19765 raft_consensus.cc:482] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:08.766744 19765 raft_consensus.cc:504] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.767565 19763 leader_election.cc:288] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a98de7a10e496ea43e0aa542a1b02d (127.0.113.2:37969), a16624c04cf042b3be5e1817ba04043c (127.0.113.3:33105)
I20260806 11:44:08.767781 19765 leader_election.cc:288] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 646a7227ca5c4fd49b9d984562b8be9d (127.0.113.1:36181), a16624c04cf042b3be5e1817ba04043c (127.0.113.3:33105)
I20260806 11:44:08.774104 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.768716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:37553 (local address 127.0.113.1:36181)
0806 11:44:08.768928 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:08.768933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:08.769066 (+ 133us) server_negotiation.cc:407] Connection header received
0806 11:44:08.769298 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:08.769302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:08.769352 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:08.769408 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:08.770110 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.770945 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.772720 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.773284 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.773357 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.773413 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:08.773839 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 11:44:08.773955 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":113}
I20260806 11:44:08.774243 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.768085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:36181 (local address 127.0.113.2:37553)
0806 11:44:08.768985 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:08.768999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:08.769161 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:08.769567 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:08.769572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:08.769594 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:08.769971 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.769983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.771067 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.771069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.772598 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.772609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.773502 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.773505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.773669 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.773695 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:44:08.773921 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 11:44:08.773987 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":184,"threads_started":1}
I20260806 11:44:08.774962 19226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b529f868e888470a82888d40b92ea651" candidate_uuid: "80a98de7a10e496ea43e0aa542a1b02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a7227ca5c4fd49b9d984562b8be9d" is_pre_election: true
I20260806 11:44:08.775264 19226 raft_consensus.cc:2454] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a98de7a10e496ea43e0aa542a1b02d in term 0.
I20260806 11:44:08.775882 19329 leader_election.cc:302] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646a7227ca5c4fd49b9d984562b8be9d, 80a98de7a10e496ea43e0aa542a1b02d; no voters:
I20260806 11:44:08.776178 19765 raft_consensus.cc:2781] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:44:08.776254 19765 raft_consensus.cc:482] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:44:08.776294 19765 raft_consensus.cc:3037] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:08.777513 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.769802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:49731 (local address 127.0.113.2:37969)
0806 11:44:08.769940 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:08.769944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:08.770277 (+ 333us) server_negotiation.cc:407] Connection header received
0806 11:44:08.771364 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:08.771367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:08.771412 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:08.771455 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:08.772100 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.772743 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.776123 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.776694 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.776770 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.776820 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:08.777293 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 11:44:08.777406 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":44}
I20260806 11:44:08.777670 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.769506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:40623 (local address 127.0.113.3:33105)
0806 11:44:08.770596 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:08.770604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:08.770967 (+ 363us) server_negotiation.cc:407] Connection header received
0806 11:44:08.771020 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:08.771026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:08.771070 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:08.771203 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:08.772833 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.773520 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.776210 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.776843 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.776938 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.776995 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:08.777416 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 11:44:08.777483 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":127,"threads_started":1}
I20260806 11:44:08.777992 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.767941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:37969 (local address 127.0.113.1:49731)
0806 11:44:08.770203 (+ 2262us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:08.770215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:08.770890 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:08.771850 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:08.771853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:08.771871 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:08.771994 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.771999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.773688 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.773691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.775952 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.775976 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.776974 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.776977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.777126 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.777150 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 11:44:08.777632 (+ 482us) client_negotiation.cc:240] Negotiation successful
0806 11:44:08.777713 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2201,"thread_start_us":120,"threads_started":1}
I20260806 11:44:08.778591 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.769130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:33105 (local address 127.0.113.2:40623)
0806 11:44:08.770693 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:08.770707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:08.770789 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:08.771550 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:08.771553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:08.771564 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:08.771677 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.771683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.774664 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.774668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.776080 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.776093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.777051 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.777054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.777232 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.777263 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 11:44:08.777330 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 11:44:08.777890 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1509,"mutex_wait_us":424,"thread_start_us":106,"threads_started":1}
I20260806 11:44:08.778784 19418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b529f868e888470a82888d40b92ea651" candidate_uuid: "646a7227ca5c4fd49b9d984562b8be9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a98de7a10e496ea43e0aa542a1b02d" is_pre_election: true
I20260806 11:44:08.778957 19765 raft_consensus.cc:504] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.779153 19418 raft_consensus.cc:2379] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 646a7227ca5c4fd49b9d984562b8be9d in current term 1: Already voted for candidate 80a98de7a10e496ea43e0aa542a1b02d in this term.
I20260806 11:44:08.779245 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b529f868e888470a82888d40b92ea651" candidate_uuid: "80a98de7a10e496ea43e0aa542a1b02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16624c04cf042b3be5e1817ba04043c" is_pre_election: true
I20260806 11:44:08.779614 19638 raft_consensus.cc:2454] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a98de7a10e496ea43e0aa542a1b02d in term 0.
I20260806 11:44:08.779785 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b529f868e888470a82888d40b92ea651" candidate_uuid: "80a98de7a10e496ea43e0aa542a1b02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16624c04cf042b3be5e1817ba04043c"
I20260806 11:44:08.779920 19637 raft_consensus.cc:3037] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:08.783066 19637 raft_consensus.cc:2454] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a98de7a10e496ea43e0aa542a1b02d in term 1.
I20260806 11:44:08.783439 19330 leader_election.cc:302] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a98de7a10e496ea43e0aa542a1b02d, a16624c04cf042b3be5e1817ba04043c; no voters:
I20260806 11:44:08.783573 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.768619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:45497 (local address 127.0.113.3:33105)
0806 11:44:08.768784 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:08.768809 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:08.778236 (+ 9427us) server_negotiation.cc:407] Connection header received
0806 11:44:08.778479 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:08.778483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:08.778529 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:08.778591 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:08.779973 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.780674 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.782187 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.782837 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.782932 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.782999 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:08.783390 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 11:44:08.783459 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 11:44:08.783933 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.768493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:33105 (local address 127.0.113.1:45497)
0806 11:44:08.778167 (+ 9674us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:08.778183 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:08.778316 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:08.778746 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:08.778751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:08.778777 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:08.778846 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.778852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.780829 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.780832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.782043 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:08.782054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.783123 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.783127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:08.783212 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.783233 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 11:44:08.783734 (+ 501us) client_negotiation.cc:240] Negotiation successful
0806 11:44:08.783805 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9607,"thread_start_us":104,"threads_started":1}
I20260806 11:44:08.784132 19765 leader_election.cc:288] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [CANDIDATE]: Term 1 election: Requested vote from peers 646a7227ca5c4fd49b9d984562b8be9d (127.0.113.1:36181), a16624c04cf042b3be5e1817ba04043c (127.0.113.3:33105)
I20260806 11:44:08.784233 19765 raft_consensus.cc:2781] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:08.784294 19765 raft_consensus.cc:686] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 1 LEADER]: Becoming Leader. State: Replica: 80a98de7a10e496ea43e0aa542a1b02d, State: Running, Role: LEADER
I20260806 11:44:08.784621 19765 consensus_queue.cc:237] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } }
I20260806 11:44:08.784643 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b529f868e888470a82888d40b92ea651" candidate_uuid: "646a7227ca5c4fd49b9d984562b8be9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16624c04cf042b3be5e1817ba04043c" is_pre_election: true
I20260806 11:44:08.784849 19637 raft_consensus.cc:2379] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 646a7227ca5c4fd49b9d984562b8be9d in current term 1: Already voted for candidate 80a98de7a10e496ea43e0aa542a1b02d in this term.
I20260806 11:44:08.784034 19226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b529f868e888470a82888d40b92ea651" candidate_uuid: "80a98de7a10e496ea43e0aa542a1b02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a7227ca5c4fd49b9d984562b8be9d"
I20260806 11:44:08.785140 19226 raft_consensus.cc:3037] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:08.785508 19126 leader_election.cc:302] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 646a7227ca5c4fd49b9d984562b8be9d; no voters: 80a98de7a10e496ea43e0aa542a1b02d, a16624c04cf042b3be5e1817ba04043c
I20260806 11:44:08.786906 19226 raft_consensus.cc:2454] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a98de7a10e496ea43e0aa542a1b02d in term 1.
I20260806 11:44:08.787076 19763 raft_consensus.cc:2726] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 11:44:08.790510 18983 catalog_manager.cc:5495] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a98de7a10e496ea43e0aa542a1b02d (127.0.113.2). New cstate: current_term: 1 leader_uuid: "80a98de7a10e496ea43e0aa542a1b02d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a98de7a10e496ea43e0aa542a1b02d" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 } health_report { overall_health: UNKNOWN } } }
I20260806 11:44:08.864355 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.859283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.0.113.2:37969)
0806 11:44:08.859489 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:08.859494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:08.859513 (+ 19us) server_negotiation.cc:407] Connection header received
0806 11:44:08.859726 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:08.859729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:08.859786 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:08.859910 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:44:08.860206 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.860958 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.862000 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:08.862213 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:08.862335 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:08.862709 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:44:08.863200 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:44:08.863324 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:08.863741 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 11:44:08.863800 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
Aug 06 11:44:08 dist-test-slave-45dx krb5kdc[17428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for HTTP/127.0.113.20@KRBTEST.COM
I20260806 11:44:08.871583 19637 raft_consensus.cc:1261] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 1 FOLLOWER]: Refusing update from remote peer 80a98de7a10e496ea43e0aa542a1b02d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:08.872391 19765 consensus_queue.cc:1035] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a16624c04cf042b3be5e1817ba04043c" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:08.874094 19226 raft_consensus.cc:1261] T b529f868e888470a82888d40b92ea651 P 646a7227ca5c4fd49b9d984562b8be9d [term 1 FOLLOWER]: Refusing update from remote peer 80a98de7a10e496ea43e0aa542a1b02d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:08.874739 19765 consensus_queue.cc:1035] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [LEADER]: Connected to new peer: Peer: permanent_uuid: "646a7227ca5c4fd49b9d984562b8be9d" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:08.886222 19785 mvcc.cc:205] Tried to move back new op lower bound from 7315524193770692608 to 7315524193431433216. Current Snapshot: MvccSnapshot[applied={T|T < 7315524193770692608}]
I20260806 11:44:08.892083 19786 mvcc.cc:205] Tried to move back new op lower bound from 7315524193770692608 to 7315524193431433216. Current Snapshot: MvccSnapshot[applied={T|T < 7315524193770692608}]
I20260806 11:44:08.892470 19788 mvcc.cc:205] Tried to move back new op lower bound from 7315524193770692608 to 7315524193431433216. Current Snapshot: MvccSnapshot[applied={T|T < 7315524193770692608}]
I20260806 11:44:08.900678 18983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:08.900858 18983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:08.903533 18983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f8de09a8b094222968bcc05cb608cde: Sending DeleteTablet for 3 replicas of tablet b529f868e888470a82888d40b92ea651
I20260806 11:44:08.904412 19408 tablet_service.cc:1514] Processing DeleteTablet for tablet b529f868e888470a82888d40b92ea651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:08 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:41164
I20260806 11:44:08.904695 19178 tablet_service.cc:1514] Processing DeleteTablet for tablet b529f868e888470a82888d40b92ea651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:08 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:57454
I20260806 11:44:08.904831 19604 tablet_service.cc:1514] Processing DeleteTablet for tablet b529f868e888470a82888d40b92ea651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:08 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:33200
I20260806 11:44:08.905622 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 19046
I20260806 11:44:08.916430 19803 tablet_replica.cc:330] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: stopping tablet replica
I20260806 11:44:08.916872 19803 raft_consensus.cc:2229] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:08.917408 19803 raft_consensus.cc:2258] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:08.917474 19798 tablet_replica.cc:330] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: stopping tablet replica
I20260806 11:44:08.917783 19798 raft_consensus.cc:2229] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:44:08.918200 19798 raft_consensus.cc:2258] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:08.919343 19803 ts_tablet_manager.cc:1905] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:08.921551 19798 ts_tablet_manager.cc:1905] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:08.922405 19803 ts_tablet_manager.cc:1918] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:08.922578 19803 log.cc:1196] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/b529f868e888470a82888d40b92ea651
I20260806 11:44:08.923300 19803 ts_tablet_manager.cc:1939] T b529f868e888470a82888d40b92ea651 P a16624c04cf042b3be5e1817ba04043c: Deleting consensus metadata
I20260806 11:44:08.924757 18955 catalog_manager.cc:4841] TS a16624c04cf042b3be5e1817ba04043c (127.0.113.3:33105): tablet b529f868e888470a82888d40b92ea651 (table test-table [id=be4e4b6380034cdd9a79e53371701ebf]) successfully deleted
I20260806 11:44:08.924278 19798 ts_tablet_manager.cc:1918] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:08.928856 19798 log.cc:1196] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/b529f868e888470a82888d40b92ea651
I20260806 11:44:08.929318 19798 ts_tablet_manager.cc:1939] T b529f868e888470a82888d40b92ea651 P 80a98de7a10e496ea43e0aa542a1b02d: Deleting consensus metadata
I20260806 11:44:08.930785 18951 catalog_manager.cc:4841] TS 80a98de7a10e496ea43e0aa542a1b02d (127.0.113.2:37969): tablet b529f868e888470a82888d40b92ea651 (table test-table [id=be4e4b6380034cdd9a79e53371701ebf]) successfully deleted
W20260806 11:44:08.932466 18953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:36181 (error 108)
I20260806 11:44:08.933491 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 19301
I20260806 11:44:08.942001 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:08.941545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:36181 (local address 127.0.0.1:57460)
0806 11:44:08.941767 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:08.941906 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260806 11:44:08.942356 18953 catalog_manager.cc:4584] TS 646a7227ca5c4fd49b9d984562b8be9d (127.0.113.1:36181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b529f868e888470a82888d40b92ea651: Network error: Client connection negotiation failed: client connection to 127.0.113.1:36181: connect: Connection refused (error 111)
I20260806 11:44:08.949561 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 19513
I20260806 11:44:08.974634 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 18919
I20260806 11:44:08.985178 452 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 11:44:09.330726 452 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 11:44:09.330848 452 mini_ranger.cc:66] Stopping Ranger...
I20260806 11:44:09.679728 452 mini_ranger.cc:68] Stopped Ranger
2026-08-06 11:44:09.679 UTC [17468] LOG: received smart shutdown request
2026-08-06 11:44:09.684 UTC [17468] LOG: background worker "logical replication launcher" (PID 17482) exited with exit code 1
2026-08-06 11:44:09.684 UTC [17474] LOG: shutting down
2026-08-06 11:44:09.837 UTC [17468] LOG: database system is shut down
2026-08-06T11:44:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49058 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:44:10 dist-test-slave-45dx krb5kdc[19821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:44:10 dist-test-slave-45dx krb5kdc[19821](info): set up 2 sockets
Aug 06 11:44:10 dist-test-slave-45dx krb5kdc[19821](info): commencing operation
krb5kdc: starting...
W20260806 11:44:11.182680 452 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:44:11.304776 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33443
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:33443,127.0.113.61:37829,127.0.113.60:44789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:11.418221 19840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:11.418486 19840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:11.418524 19840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:11.422646 19840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:11.422712 19840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:11.422734 19840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:11.422749 19840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:11.422765 19840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:11.428089 19840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:33443,127.0.113.61:37829,127.0.113.60:44789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:11.429322 19840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:11.430433 19840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:11.439644 19854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:11.439687 19840 server_base.cc:1034] running on GCE node
W20260806 11:44:11.439647 19848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:11.439647 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:11.440620 19840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:11.441880 19840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:11.443076 19840 hybrid_clock.cc:648] HybridClock initialized: now 1786016651443051 us; error 39 us; skew 500 ppm
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:11.446038 19840 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:44:11.446936 19840 webserver.cc:460] Webserver started at http://127.0.113.62:33771/ using document root <none> and password file <none>
I20260806 11:44:11.447495 19840 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:11.447557 19840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:11.447790 19840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:11.449892 19840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "925ae34ed3d24abcbaeed309f82271cf"
format_stamp: "Formatted at 2026-08-06 11:44:11 on dist-test-slave-45dx"
server_key: "21bd8f280a57a7bafd60137d898164e5"
server_key_iv: "d87c6755ce94e855ff2910da7a8a229b"
server_key_version: "encryptionkey@0"
I20260806 11:44:11.450481 19840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "925ae34ed3d24abcbaeed309f82271cf"
format_stamp: "Formatted at 2026-08-06 11:44:11 on dist-test-slave-45dx"
server_key: "21bd8f280a57a7bafd60137d898164e5"
server_key_iv: "d87c6755ce94e855ff2910da7a8a229b"
server_key_version: "encryptionkey@0"
I20260806 11:44:11.454263 19840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:44:11.456665 19865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:11.457866 19840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:44:11.457995 19840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "925ae34ed3d24abcbaeed309f82271cf"
format_stamp: "Formatted at 2026-08-06 11:44:11 on dist-test-slave-45dx"
server_key: "21bd8f280a57a7bafd60137d898164e5"
server_key_iv: "d87c6755ce94e855ff2910da7a8a229b"
server_key_version: "encryptionkey@0"
I20260806 11:44:11.458099 19840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:11.475476 19840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:11.478380 19840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:11.478583 19840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:11.485713 19840 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:33443
I20260806 11:44:11.485728 19947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:33443 every 8 connection(s)
I20260806 11:44:11.486625 19840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:44:11.489308 19948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:11.491008 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 19840
I20260806 11:44:11.491142 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:44:11.491425 452 external_mini_cluster.cc:1428] Setting key 0b97a502207d8d90d74a3957a3ab4ecf
I20260806 11:44:11.494184 19948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.61@KRBTEST.COM" created.
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:11.509305 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.496485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33443 (local address 127.0.0.1:55964)
0806 11:44:11.497019 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.497054 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:11.497828 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:11.498464 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:11.498469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:11.498913 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:11.499150 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.499164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.500330 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.500334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.501091 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.501100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.501856 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.501860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.501966 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.502526 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:11.502542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:11.504246 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:11.506223 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.506235 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.506249 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.506584 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.506955 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.506961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.506966 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.507130 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.507728 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:11.507737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:11.507927 (+ 190us) client_negotiation.cc:769] Sending connection context
0806 11:44:11.508120 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 11:44:11.508587 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":98,"thread_start_us":164,"threads_started":1}
I20260806 11:44:11.509303 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.496486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.0.113.62:33443)
0806 11:44:11.496882 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.496901 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.497154 (+ 253us) server_negotiation.cc:407] Connection header received
0806 11:44:11.498122 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.498127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.498286 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.498464 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.499310 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.500182 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.501211 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.501664 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.501884 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.504442 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.504455 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.504464 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.504484 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.506059 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.506718 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.506723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.506724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.506851 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.507235 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.507238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.507239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.507601 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.507700 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.508198 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.508472 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260806 11:44:11.511982 19948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:44:11.512765 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.512461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:37829 (local address 127.0.0.1:42184)
0806 11:44:11.512592 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.512690 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 11:44:11.513002 19867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
I20260806 11:44:11.513908 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.513692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:37829 (local address 127.0.0.1:42198)
0806 11:44:11.513789 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.513861 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 11:44:11.514106 19948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.61:37829: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab.
Entry for principal kudu/127.0.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab.
I20260806 11:44:11.520903 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.61:37829
--webserver_interface=127.0.113.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab
--principal=kudu/127.0.113.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:33443,127.0.113.61:37829,127.0.113.60:44789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:44:11.563349 19948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } attempt: 1
I20260806 11:44:11.564343 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.564000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:37829 (local address 127.0.0.1:42200)
0806 11:44:11.564153 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.564219 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 11:44:11.565109 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.564878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:37829 (local address 127.0.0.1:42206)
0806 11:44:11.564990 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.565049 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 11:44:11.565243 19948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.61:37829: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
I20260806 11:44:11.633424 19948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } attempt: 2
I20260806 11:44:11.634321 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.634018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:37829 (local address 127.0.0.1:42218)
0806 11:44:11.634168 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.634243 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 11:44:11.634512 19959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:11.634812 19959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:11.634919 19959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 11:44:11.635231 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.634951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:37829 (local address 127.0.0.1:42230)
0806 11:44:11.635102 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.635160 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 11:44:11.635419 19948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.61:37829: Network error: Client connection negotiation failed: client connection to 127.0.113.61:37829: connect: Connection refused (error 111)
W20260806 11:44:11.639230 19959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:11.639315 19959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:11.639364 19959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:11.639395 19959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:11.639427 19959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:11.644634 19959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:33443,127.0.113.61:37829,127.0.113.60:44789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab
--principal=kudu/127.0.113.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.61:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.113.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:11.645674 19959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:11.646723 19959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:11.655468 19969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:11.655411 19968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:11.655411 19973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:11.655537 19959 server_base.cc:1034] running on GCE node
I20260806 11:44:11.656286 19959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:11.657618 19959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:11.658823 19959 hybrid_clock.cc:648] HybridClock initialized: now 1786016651658809 us; error 47 us; skew 500 ppm
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:11.662706 19959 init.cc:377] Logged in from keytab as kudu/127.0.113.61@KRBTEST.COM (short username kudu)
I20260806 11:44:11.663653 19959 webserver.cc:460] Webserver started at http://127.0.113.61:35141/ using document root <none> and password file <none>
I20260806 11:44:11.664250 19959 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:11.664378 19959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:11.664613 19959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:11.666714 19959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/data/instance:
uuid: "0f66bd6a845d468c914142361c51716a"
format_stamp: "Formatted at 2026-08-06 11:44:11 on dist-test-slave-45dx"
server_key: "e3443213b0edfbc9e7a50b060be2d9bc"
server_key_iv: "a49cd5b6f09406d03bc4fa59e0d3e459"
server_key_version: "encryptionkey@0"
I20260806 11:44:11.667309 19959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/wal/instance:
uuid: "0f66bd6a845d468c914142361c51716a"
format_stamp: "Formatted at 2026-08-06 11:44:11 on dist-test-slave-45dx"
server_key: "e3443213b0edfbc9e7a50b060be2d9bc"
server_key_iv: "a49cd5b6f09406d03bc4fa59e0d3e459"
server_key_version: "encryptionkey@0"
I20260806 11:44:11.671021 19959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 11:44:11.673487 19986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:11.674929 19959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 11:44:11.675076 19959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/wal
uuid: "0f66bd6a845d468c914142361c51716a"
format_stamp: "Formatted at 2026-08-06 11:44:11 on dist-test-slave-45dx"
server_key: "e3443213b0edfbc9e7a50b060be2d9bc"
server_key_iv: "a49cd5b6f09406d03bc4fa59e0d3e459"
server_key_version: "encryptionkey@0"
I20260806 11:44:11.675194 19959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:11.685323 19959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:11.688442 19959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:11.688644 19959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:11.695905 19959 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.61:37829
I20260806 11:44:11.695937 20067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.61:37829 every 8 connection(s)
I20260806 11:44:11.696960 19959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
I20260806 11:44:11.697732 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 19959
I20260806 11:44:11.697863 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-1/wal/instance
I20260806 11:44:11.698174 452 external_mini_cluster.cc:1428] Setting key c96e18399ac7d1e3cd8f212c21c8f396
I20260806 11:44:11.700518 20068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:11.704437 20068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.113.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.60@KRBTEST.COM" created.
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:11.719024 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.706334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.0.113.62:33443)
0806 11:44:11.706548 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.706553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.707338 (+ 785us) server_negotiation.cc:407] Connection header received
0806 11:44:11.708421 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.708424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.708476 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.708564 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.710018 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.710541 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.712088 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.712517 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.712655 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.715195 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.715201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.715208 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.715235 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.716810 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.717369 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.717372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.717373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.717452 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.717802 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.717806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.717807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.718008 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.718115 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.718660 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.718800 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 11:44:11.719923 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.706671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33443 (local address 127.0.0.1:55966)
0806 11:44:11.707168 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.707221 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:11.708167 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:11.708744 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:11.708755 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:11.709202 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:11.709824 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.709839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.710683 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.710690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.711953 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.711964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.712712 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.712716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.712869 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.713531 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:11.713547 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:11.715016 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:11.716946 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.716953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.716968 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.717241 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.717549 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.717552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.717554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.717701 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.718129 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:11.718137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:11.718407 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 11:44:11.718601 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 11:44:11.718951 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":154,"threads_started":1}
I20260806 11:44:11.721396 20068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab.
Entry for principal kudu/127.0.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab.
I20260806 11:44:11.726593 19948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } attempt: 3
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
I20260806 11:44:11.727854 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.60:44789
--webserver_interface=127.0.113.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab
--principal=kudu/127.0.113.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:33443,127.0.113.61:37829,127.0.113.60:44789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
I20260806 11:44:11.733397 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.721967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:37829 (local address 127.0.0.1:42238)
0806 11:44:11.722093 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.722110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:11.722197 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:11.722887 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:11.722889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:11.723039 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:11.723128 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.723133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.724242 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.724244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.724888 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.724898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.725630 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.725634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.725709 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.726224 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:11.726236 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:11.727670 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:11.731581 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.731584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.731586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.731800 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.732157 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.732162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.732164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.732240 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.732829 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:11.732832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:11.732923 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 11:44:11.733047 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 11:44:11.733181 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 11:44:11.733433 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.722101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.0.113.61:37829)
0806 11:44:11.722434 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.722449 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.722473 (+ 24us) server_negotiation.cc:407] Connection header received
0806 11:44:11.722542 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.722550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.722729 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.722838 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.723252 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.724102 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.725061 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.725459 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.725594 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.728751 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.728761 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.728773 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.728795 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.731456 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.731941 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.731948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.731949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.732051 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.732353 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.732356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.732358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.732712 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.732860 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.733138 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.733268 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":161,"threads_started":1}
I20260806 11:44:11.735656 20068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:44:11.736552 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.736162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51816)
0806 11:44:11.736337 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.736439 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 11:44:11.736760 19991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
I20260806 11:44:11.738135 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.727177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.0.113.61:37829)
0806 11:44:11.727516 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.727531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.727547 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:44:11.727584 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.727590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.727654 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.727754 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.728730 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.729582 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.730792 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.731178 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.731294 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.733915 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.733922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.733928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.733959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.735924 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.736722 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.736726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.736728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.736821 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.737178 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.737181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.737183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.737445 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.737557 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.737915 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.738016 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
I20260806 11:44:11.738245 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.727114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:37829 (local address 127.0.0.1:42240)
0806 11:44:11.727246 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.727262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:11.727346 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:11.727807 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:11.727810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:11.728005 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:11.728605 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.728614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.729741 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.729744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.730645 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.730662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.731323 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.731328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.731458 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.731964 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:11.731979 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:11.733720 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:11.736344 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.736351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.736354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.736575 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.736953 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.736956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.736958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.737047 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.737574 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:11.737580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:11.737715 (+ 135us) client_negotiation.cc:769] Sending connection context
0806 11:44:11.737841 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 11:44:11.738013 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 11:44:11.738758 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.737532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51824)
0806 11:44:11.738608 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.738674 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":990}
I20260806 11:44:11.738925 19948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } has no permanent_uuid. Determining permanent_uuid...
W20260806 11:44:11.738917 20068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:44789: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
I20260806 11:44:11.739711 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.739390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51834)
0806 11:44:11.739553 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.739621 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 11:44:11.740502 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.740173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51850)
0806 11:44:11.740357 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.740412 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260806 11:44:11.740644 19948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:44789: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
I20260806 11:44:11.771847 19948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } attempt: 1
I20260806 11:44:11.772912 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.772537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51864)
0806 11:44:11.772722 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.772805 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 11:44:11.773748 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.773481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51880)
0806 11:44:11.773597 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.773663 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 11:44:11.773905 19948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:44789: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
I20260806 11:44:11.788231 20068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } attempt: 1
I20260806 11:44:11.789229 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.788864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51882)
0806 11:44:11.789049 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.789129 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 11:44:11.790009 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.789770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51890)
0806 11:44:11.789891 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.789945 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 11:44:11.790164 20068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:44789: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
I20260806 11:44:11.849083 19948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } attempt: 2
I20260806 11:44:11.849984 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.849641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51898)
0806 11:44:11.849814 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.849889 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 11:44:11.850701 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.850496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51912)
0806 11:44:11.850594 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.850644 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 11:44:11.850881 19948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:44789: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
W20260806 11:44:11.851542 20080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:11.851778 20080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:11.851873 20080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:11.856007 20080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:11.856057 20080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:11.856077 20080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:11.856094 20080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:11.856144 20080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:11.858372 20068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } attempt: 2
I20260806 11:44:11.859157 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.858856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51914)
0806 11:44:11.859011 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.859075 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 11:44:11.859874 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.859661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51920)
0806 11:44:11.859765 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.859815 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 11:44:11.860042 20068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:44789: Network error: Client connection negotiation failed: client connection to 127.0.113.60:44789: connect: Connection refused (error 111)
I20260806 11:44:11.861989 20080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:33443,127.0.113.61:37829,127.0.113.60:44789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab
--principal=kudu/127.0.113.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.60:44789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.113.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:11.863066 20080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:11.864118 20080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:11.872689 20089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:11.872733 20090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:11.872844 20080 server_base.cc:1034] running on GCE node
W20260806 11:44:11.872733 20094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:11.873624 20080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:11.874776 20080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:11.875969 20080 hybrid_clock.cc:648] HybridClock initialized: now 1786016651875950 us; error 36 us; skew 500 ppm
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:11.878815 20080 init.cc:377] Logged in from keytab as kudu/127.0.113.60@KRBTEST.COM (short username kudu)
I20260806 11:44:11.879794 20080 webserver.cc:460] Webserver started at http://127.0.113.60:44093/ using document root <none> and password file <none>
I20260806 11:44:11.880489 20080 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:11.880586 20080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:11.880797 20080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:11.882835 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/data/instance:
uuid: "346cb5772e7b4125b08b093751917b22"
format_stamp: "Formatted at 2026-08-06 11:44:11 on dist-test-slave-45dx"
server_key: "ae72266473d812c0f8bd4c0ddc7c9773"
server_key_iv: "689f46848e9bde6a036c9244e55e9aaf"
server_key_version: "encryptionkey@0"
I20260806 11:44:11.883476 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/wal/instance:
uuid: "346cb5772e7b4125b08b093751917b22"
format_stamp: "Formatted at 2026-08-06 11:44:11 on dist-test-slave-45dx"
server_key: "ae72266473d812c0f8bd4c0ddc7c9773"
server_key_iv: "689f46848e9bde6a036c9244e55e9aaf"
server_key_version: "encryptionkey@0"
I20260806 11:44:11.887199 20080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260806 11:44:11.889637 20108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:11.890851 20080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:44:11.890992 20080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/wal
uuid: "346cb5772e7b4125b08b093751917b22"
format_stamp: "Formatted at 2026-08-06 11:44:11 on dist-test-slave-45dx"
server_key: "ae72266473d812c0f8bd4c0ddc7c9773"
server_key_iv: "689f46848e9bde6a036c9244e55e9aaf"
server_key_version: "encryptionkey@0"
I20260806 11:44:11.891105 20080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:11.906944 20080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:11.909857 20080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:11.910087 20080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:11.917079 20080 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.60:44789
I20260806 11:44:11.917109 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.60:44789 every 8 connection(s)
I20260806 11:44:11.918015 20080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
I20260806 11:44:11.920714 20185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:11.924357 20185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:44:11.926748 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 20080
I20260806 11:44:11.926899 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786016488822243-452-0/minicluster-data/master-2/wal/instance
I20260806 11:44:11.927205 452 external_mini_cluster.cc:1428] Setting key 84580c4e59f238ead2976627f656bd59
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
I20260806 11:44:11.938594 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.926185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.0.113.62:33443)
0806 11:44:11.926334 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.926340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.927351 (+ 1011us) server_negotiation.cc:407] Connection header received
0806 11:44:11.928554 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.928556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.928601 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.928666 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.929709 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.930199 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.931308 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.931533 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.931643 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.934399 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.934403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.934406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.934442 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.936144 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.936790 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.936792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.936794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.936889 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.937236 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.937240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.937244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.937533 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.937642 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.938267 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.938413 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 11:44:11.939565 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.926640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:33443 (local address 127.0.0.1:55968)
0806 11:44:11.927188 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.927229 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:11.928332 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:11.928754 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:11.928764 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:11.929218 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:11.929579 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.929599 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.930305 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.930308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.931182 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.931190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.931693 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.931697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.931920 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.932585 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:11.932609 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:11.934202 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:11.936256 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.936266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.936277 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.936698 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.936975 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.936977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.936978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.937117 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.937656 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:11.937665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:11.937985 (+ 320us) client_negotiation.cc:769] Sending connection context
0806 11:44:11.938179 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 11:44:11.938518 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":74,"spinlock_wait_cycles":2816,"thread_start_us":152,"threads_started":1}
I20260806 11:44:11.939579 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.927895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.0.113.60:44789)
0806 11:44:11.928275 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.928286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.928381 (+ 95us) server_negotiation.cc:407] Connection header received
0806 11:44:11.928517 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.928525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.928754 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.928929 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.929620 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.930635 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.931449 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.931844 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.931974 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.934858 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.934863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.934874 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.934895 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.936692 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.937322 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.937326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.937327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.937395 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.937678 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.937681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.937682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.937977 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.938185 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.938384 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.938567 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260806 11:44:11.940928 20185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:11.950022 19948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } attempt: 3
I20260806 11:44:11.950608 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.941534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42250 (local address 127.0.113.61:37829)
0806 11:44:11.941643 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.941646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.941851 (+ 205us) server_negotiation.cc:407] Connection header received
0806 11:44:11.942016 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.942020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.942080 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.942140 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.942789 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.943514 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.944578 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.944885 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.944986 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.947137 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.947141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.947143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.947166 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.948850 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.949286 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.949290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.949292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.949387 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.949669 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.949674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.949676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.949863 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.949970 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.950316 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.950449 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 11:44:11.950906 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.942009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.0.113.62:33443)
0806 11:44:11.942113 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.942121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.942133 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:44:11.942230 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.942232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.942266 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.942327 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.943057 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.943728 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.944761 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.945079 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.945167 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.947643 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.947647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.947649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.947670 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.949039 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.949461 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.949465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.949467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.949549 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.949836 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.949839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.949841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.950036 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.950322 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.950400 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.950516 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 11:44:11.951133 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.941451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:37829 (local address 127.0.0.1:42250)
0806 11:44:11.941756 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.941769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:11.941877 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:11.942301 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:11.942323 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:11.942512 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:11.942653 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.942662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.943627 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.943629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.944463 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.944473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.945011 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.945014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.945095 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.945569 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:11.945581 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:11.946984 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:11.948967 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.948972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.948974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.949180 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.949482 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.949485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.949487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.949577 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.949962 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:11.949965 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:11.950081 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 11:44:11.950811 (+ 730us) client_negotiation.cc:240] Negotiation successful
0806 11:44:11.950960 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243}
I20260806 11:44:11.951243 20068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } attempt: 3
I20260806 11:44:11.952744 20185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
Aug 06 11:44:11 dist-test-slave-45dx krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
I20260806 11:44:11.961593 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.950443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51932)
0806 11:44:11.950723 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.950740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:11.950839 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:11.951943 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:11.951949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:11.952119 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:11.952212 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.952217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.953255 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.953261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.954082 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.954092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.954774 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.954777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.954867 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.955386 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:11.955399 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:11.956792 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:11.959845 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.959848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.959850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.960105 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.960525 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.960531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.960536 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.960630 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.961065 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:11.961069 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:11.961175 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 11:44:11.961261 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 11:44:11.961370 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260806 11:44:11.962538 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.950981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.0.113.60:44789)
0806 11:44:11.951742 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.951746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.951759 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:44:11.951792 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.951794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.951837 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.951925 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.952376 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.953130 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.954217 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.954641 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.955114 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.956927 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.956931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.956933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.956956 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.959708 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.960277 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.960283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.960286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.960405 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.960736 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.960738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.960739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.960944 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.962140 (+ 1196us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.962215 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.962335 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":688}
I20260806 11:44:11.963840 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.953182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51948)
0806 11:44:11.953621 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.953635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:11.953719 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:11.954125 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:11.954128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:11.954285 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:11.954367 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.954373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.955422 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.955425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.956208 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.956217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.957067 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.957073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.957222 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.957628 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:11.957649 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:11.959150 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:11.961934 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.961938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.961943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.962212 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.962522 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.962526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.962528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.962648 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.963332 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:11.963338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:11.963432 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 11:44:11.963534 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 11:44:11.963657 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":22,"spinlock_wait_cycles":9088}
I20260806 11:44:11.964475 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.953398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.0.113.60:44789)
0806 11:44:11.953928 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.953937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.953948 (+ 11us) server_negotiation.cc:407] Connection header received
0806 11:44:11.953979 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.953981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.954015 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.954457 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.954558 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.955307 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.956522 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.956958 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.957061 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.959280 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.959284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.959286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.959316 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.961816 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.962315 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.962319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.962321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.962411 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.962756 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.962760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.962762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.962956 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.964095 (+ 1139us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.964176 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.964281 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":467,"spinlock_wait_cycles":1792,"thread_start_us":117,"threads_started":1}
I20260806 11:44:11.964617 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.952604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.0.113.61:37829)
0806 11:44:11.952707 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.952710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.952772 (+ 62us) server_negotiation.cc:407] Connection header received
0806 11:44:11.952999 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.953002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.953040 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.953082 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.953912 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.954614 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.955730 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.956254 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.956389 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.960635 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.960640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.960643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.960670 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.962333 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.962803 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.962807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.962808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.962919 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.963255 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.963258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.963260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.963483 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.963710 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.964293 (+ 583us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.964461 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":195,"server-negotiator.queue_time_us":41}
I20260806 11:44:11.965327 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.951634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:44789 (local address 127.0.0.1:51944)
0806 11:44:11.951732 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:11.951747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:11.951814 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:11.952924 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:11.952927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:11.953123 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:11.953246 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.953253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.954866 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.954869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.956060 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:11.956070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.957389 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.957394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:11.957491 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.957873 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:11.957886 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:11.959822 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:11.963367 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.963370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.963372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.963621 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.963921 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:11.963924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:11.963925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:11.964037 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:11.964822 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:11.964825 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:11.964930 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 11:44:11.965040 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 11:44:11.965168 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260806 11:44:11.965903 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.952062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.0.113.60:44789)
0806 11:44:11.952482 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.952492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.952505 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:44:11.952539 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.952732 (+ 193us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.952783 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.953356 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.953454 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.954747 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.956522 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.957259 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.957712 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.959984 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.960002 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.960006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.960034 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:11.963246 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.963736 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.963739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.963741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.963815 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:11.964147 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:11.964149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:11.964151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:11.964636 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:11.965580 (+ 944us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:11.965650 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 11:44:11.965749 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":330,"spinlock_wait_cycles":1792,"thread_start_us":129,"threads_started":1}
I20260806 11:44:11.966622 19948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf: Bootstrap starting.
I20260806 11:44:11.966974 20185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22: Bootstrap starting.
I20260806 11:44:11.969043 20185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:11.969134 19948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:11.969751 20185 log.cc:824] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:11.969965 19948 log.cc:824] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:11.970754 20068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a: Bootstrap starting.
I20260806 11:44:11.971711 19948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf: No bootstrap required, opened a new log
I20260806 11:44:11.971848 20185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22: No bootstrap required, opened a new log
I20260806 11:44:11.973297 20068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:11.974120 20068 log.cc:824] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:11.975286 19948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } }
I20260806 11:44:11.975507 19948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:11.975571 19948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925ae34ed3d24abcbaeed309f82271cf, State: Initialized, Role: FOLLOWER
I20260806 11:44:11.976066 19948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } }
I20260806 11:44:11.976517 20068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a: No bootstrap required, opened a new log
I20260806 11:44:11.976450 20185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } }
I20260806 11:44:11.976675 20185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:11.976722 20185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346cb5772e7b4125b08b093751917b22, State: Initialized, Role: FOLLOWER
I20260806 11:44:11.977154 20185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } }
I20260806 11:44:11.977486 19948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:11.978380 20185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 11:44:11.980058 19877 tablet.cc:2369] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:44:11.980099 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:44:11.981958 20068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } }
I20260806 11:44:11.982191 20068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:11.982240 20068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f66bd6a845d468c914142361c51716a, State: Initialized, Role: FOLLOWER
I20260806 11:44:11.982673 20068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } }
W20260806 11:44:11.983740 20233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:44:11.983741 20234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:44:11.983819 20233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 11:44:11.983819 20234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:44:11.983896 20068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:11.986919 20201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:11.987135 20201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:44:11.987392 20203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:11.987596 20203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:44:11.984751 19840 master_runner.cc:186] Error getting master registration for 127.0.113.61:37829: OK, instance_id { permanent_uuid: "0f66bd6a845d468c914142361c51716a" instance_seqno: 1786016651693822 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 11:44:11.989009 19840 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 11:44:11.992935 20260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:44:11.993024 20260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:44:11.993168 20080 master_runner.cc:263] No leader master found from master 346cb5772e7b4125b08b093751917b22
I20260806 11:44:11.993230 20080 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 11:44:11.995303 19959 master_runner.cc:263] No leader master found from master 0f66bd6a845d468c914142361c51716a
I20260806 11:44:11.995362 19959 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 11:44:11.998275 20243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:11.998416 20243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:44:12.003616 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.985796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.0.113.60:44789)
0806 11:44:11.985929 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.985935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.993698 (+ 7763us) server_negotiation.cc:407] Connection header received
0806 11:44:11.993882 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.993884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.993929 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.993980 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.995021 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.995724 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.996793 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.997110 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.997170 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:11.999472 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:11.999476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:11.999479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:11.999512 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:12.001446 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:12.002012 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:12.002017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:12.002019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:12.002116 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:12.002475 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:12.002479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:12.002482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:12.002735 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:12.002874 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:12.003259 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 11:44:12.003410 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 11:44:12.004526 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.988386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.0.113.62:33443)
0806 11:44:11.988601 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.988605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.992718 (+ 4113us) server_negotiation.cc:407] Connection header received
0806 11:44:11.992920 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.992923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.992975 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.993043 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.994139 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.994935 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.996173 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.996542 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:11.996641 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:12.000908 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:12.000912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:12.000915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:12.000943 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:12.002624 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:12.003139 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:12.003142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:12.003144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:12.003229 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:12.003525 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:12.003528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:12.003530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:12.003741 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:12.003836 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:12.004206 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 11:44:12.004364 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260806 11:44:12.006601 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:11.986602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.0.113.61:37829)
0806 11:44:11.986701 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:11.986705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:11.996569 (+ 9864us) server_negotiation.cc:407] Connection header received
0806 11:44:11.996740 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:11.996744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:11.996789 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:11.996839 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:11.997801 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:11.998469 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:12.000057 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:12.000736 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:12.000856 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:12.002968 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:12.002973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:12.002975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:12.003001 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:12.004752 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:12.005239 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:12.005245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:12.005249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:12.005334 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:12.005665 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:12.005670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:12.005671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:12.005894 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:12.006017 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:12.006316 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 11:44:12.006441 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 11:44:12.111918 20243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:12.112131 20243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } }
I20260806 11:44:12.113215 20243 leader_election.cc:288] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 925ae34ed3d24abcbaeed309f82271cf (127.0.113.62:33443), 346cb5772e7b4125b08b093751917b22 (127.0.113.60:44789)
I20260806 11:44:12.113960 19905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f66bd6a845d468c914142361c51716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ae34ed3d24abcbaeed309f82271cf" is_pre_election: true
I20260806 11:44:12.114287 19905 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f66bd6a845d468c914142361c51716a in term 0.
I20260806 11:44:12.114851 20147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f66bd6a845d468c914142361c51716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346cb5772e7b4125b08b093751917b22" is_pre_election: true
I20260806 11:44:12.115139 20147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f66bd6a845d468c914142361c51716a in term 0.
I20260806 11:44:12.115645 19991 leader_election.cc:302] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f66bd6a845d468c914142361c51716a, 346cb5772e7b4125b08b093751917b22; no voters:
I20260806 11:44:12.116164 20243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:44:12.116254 20243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:44:12.116329 20243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:12.117810 20243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } }
I20260806 11:44:12.118167 20243 leader_election.cc:288] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [CANDIDATE]: Term 1 election: Requested vote from peers 925ae34ed3d24abcbaeed309f82271cf (127.0.113.62:33443), 346cb5772e7b4125b08b093751917b22 (127.0.113.60:44789)
I20260806 11:44:12.118799 20147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f66bd6a845d468c914142361c51716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346cb5772e7b4125b08b093751917b22"
I20260806 11:44:12.118953 20147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:12.120436 19905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f66bd6a845d468c914142361c51716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ae34ed3d24abcbaeed309f82271cf"
I20260806 11:44:12.120582 19905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:12.121866 19905 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f66bd6a845d468c914142361c51716a in term 1.
I20260806 11:44:12.122222 19991 leader_election.cc:302] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f66bd6a845d468c914142361c51716a, 925ae34ed3d24abcbaeed309f82271cf; no voters:
I20260806 11:44:12.120256 20147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f66bd6a845d468c914142361c51716a in term 1.
I20260806 11:44:12.124099 20243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:12.124401 20243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [term 1 LEADER]: Becoming Leader. State: Replica: 0f66bd6a845d468c914142361c51716a, State: Running, Role: LEADER
I20260806 11:44:12.124759 20243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } }
I20260806 11:44:12.126631 20243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f66bd6a845d468c914142361c51716a. Latest consensus state: current_term: 1 leader_uuid: "0f66bd6a845d468c914142361c51716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:12.126777 20243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:12.130074 20268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:44:12.131167 20268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:44:12.138481 20147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [term 1 FOLLOWER]: Refusing update from remote peer 0f66bd6a845d468c914142361c51716a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:12.139225 20243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [LEADER]: Connected to new peer: Peer: permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:12.143752 20203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f66bd6a845d468c914142361c51716a. Latest consensus state: current_term: 1 leader_uuid: "0f66bd6a845d468c914142361c51716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:12.143945 20203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:44:12.145228 19905 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [term 1 FOLLOWER]: Refusing update from remote peer 0f66bd6a845d468c914142361c51716a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:12.148511 20243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [LEADER]: Connected to new peer: Peer: permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:12.152725 20201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f66bd6a845d468c914142361c51716a. Latest consensus state: current_term: 1 leader_uuid: "0f66bd6a845d468c914142361c51716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:12.152858 20201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:44:12.158767 20266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f66bd6a845d468c914142361c51716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:12.158926 20266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:12.160252 20201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f66bd6a845d468c914142361c51716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:12.160434 20201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:44:12.161144 20269 mvcc.cc:205] Tried to move back new op lower bound from 7315524207150809088 to 7315524207106560000. Current Snapshot: MvccSnapshot[applied={T|T < 7315524207150809088}]
W20260806 11:44:12.161432 20114 tablet.cc:2369] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:44:12.162043 20272 mvcc.cc:205] Tried to move back new op lower bound from 7315524207150809088 to 7315524207106560000. Current Snapshot: MvccSnapshot[applied={T|T < 7315524207150809088}]
I20260806 11:44:12.163254 20243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f66bd6a845d468c914142361c51716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:12.163385 20243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:12.164415 20203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f66bd6a845d468c914142361c51716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925ae34ed3d24abcbaeed309f82271cf" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33443 } } peers { permanent_uuid: "0f66bd6a845d468c914142361c51716a" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 37829 } } peers { permanent_uuid: "346cb5772e7b4125b08b093751917b22" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 44789 } } }
I20260806 11:44:12.164539 20203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:44:12.169097 20270 mvcc.cc:205] Tried to move back new op lower bound from 7315524207150809088 to 7315524207106560000. Current Snapshot: MvccSnapshot[applied={T|T < 7315524207150809088}]
I20260806 11:44:12.172758 20268 catalog_manager.cc:1347] Generated new cluster ID: b1182e6875da459397ad3caaf4fb83fd
I20260806 11:44:12.172832 20268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:44:12.195768 20268 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:44:12.196844 20268 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:44:12.208719 20268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f66bd6a845d468c914142361c51716a: Generated new TSK 0
I20260806 11:44:12.209578 20268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:44:12.985208 20234 catalog_manager.cc:1259] Loaded cluster ID: b1182e6875da459397ad3caaf4fb83fd
I20260806 11:44:12.985201 20233 catalog_manager.cc:1259] Loaded cluster ID: b1182e6875da459397ad3caaf4fb83fd
I20260806 11:44:12.985299 20233 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22: loading cluster ID for follower catalog manager: success
I20260806 11:44:12.985302 20234 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf: loading cluster ID for follower catalog manager: success
I20260806 11:44:12.986956 20233 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22: acquiring CA information for follower catalog manager: success
I20260806 11:44:12.986958 20234 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf: acquiring CA information for follower catalog manager: success
I20260806 11:44:12.988091 20233 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 346cb5772e7b4125b08b093751917b22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 11:44:12.988091 20234 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 925ae34ed3d24abcbaeed309f82271cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 11:44:13.280159 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 19840
I20260806 11:44:13.296716 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 19959
I20260806 11:44:13.308135 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 20080
2026-08-06T11:44:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3352 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:44:13 dist-test-slave-45dx krb5kdc[20293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:44:13 dist-test-slave-45dx krb5kdc[20293](info): set up 2 sockets
Aug 06 11:44:13 dist-test-slave-45dx krb5kdc[20293](info): commencing operation
krb5kdc: starting...
W20260806 11:44:14.537355 452 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:44:14 dist-test-slave-45dx krb5kdc[20293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:44:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:44:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:44:14.671464 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:34257
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:45453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:34257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:14.784791 20318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:14.785054 20318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:14.785094 20318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:14.789245 20318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:14.789301 20318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:14.789320 20318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:14.789373 20318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:14.789425 20318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:14.794586 20318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:45453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:34257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:14.795590 20318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:14.796608 20318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:14.804934 20327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:14.805004 20325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:14.804981 20318 server_base.cc:1034] running on GCE node
W20260806 11:44:14.804935 20324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:14.805831 20318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:14.806861 20318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:14.808039 20318 hybrid_clock.cc:648] HybridClock initialized: now 1786016654808019 us; error 31 us; skew 500 ppm
Aug 06 11:44:14 dist-test-slave-45dx krb5kdc[20293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:14.810827 20318 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:44:14.811674 20318 webserver.cc:460] Webserver started at http://127.0.113.62:40025/ using document root <none> and password file <none>
I20260806 11:44:14.812212 20318 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:14.812305 20318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:14.812556 20318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:14.814484 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "a4d5ddebee2a4392af388b67c89b4e55"
format_stamp: "Formatted at 2026-08-06 11:44:14 on dist-test-slave-45dx"
server_key: "59bbe34bd2af7971b6845088c2c4e153"
server_key_iv: "a590e0b9d1f9073ec367d53b4a3b35ed"
server_key_version: "encryptionkey@0"
I20260806 11:44:14.815048 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "a4d5ddebee2a4392af388b67c89b4e55"
format_stamp: "Formatted at 2026-08-06 11:44:14 on dist-test-slave-45dx"
server_key: "59bbe34bd2af7971b6845088c2c4e153"
server_key_iv: "a590e0b9d1f9073ec367d53b4a3b35ed"
server_key_version: "encryptionkey@0"
I20260806 11:44:14.818672 20318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 11:44:14.821046 20338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:14.822220 20318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:44:14.822360 20318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "a4d5ddebee2a4392af388b67c89b4e55"
format_stamp: "Formatted at 2026-08-06 11:44:14 on dist-test-slave-45dx"
server_key: "59bbe34bd2af7971b6845088c2c4e153"
server_key_iv: "a590e0b9d1f9073ec367d53b4a3b35ed"
server_key_version: "encryptionkey@0"
I20260806 11:44:14.822513 20318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:14.845342 20318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:14.848644 20318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:14.848853 20318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:14.855803 20318 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:34257
I20260806 11:44:14.855821 20417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:34257 every 8 connection(s)
I20260806 11:44:14.856755 20318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:44:14.859225 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 20318
I20260806 11:44:14.859350 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:44:14.859623 20419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:14.859711 452 external_mini_cluster.cc:1428] Setting key 7391c961f885535b9cae7aa2e8eecb79
I20260806 11:44:14.866685 20419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55: Bootstrap starting.
I20260806 11:44:14.868989 20419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55: Neither blocks nor log segments found. Creating new log.
Aug 06 11:44:14 dist-test-slave-45dx krb5kdc[20293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:14.869904 20419 log.cc:824] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:14.871723 20419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55: No bootstrap required, opened a new log
I20260806 11:44:14.874729 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:14.861015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.0.113.62:34257)
0806 11:44:14.861413 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:14.861434 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:14.861468 (+ 34us) server_negotiation.cc:407] Connection header received
0806 11:44:14.862389 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:14.862408 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:14.862744 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:14.863025 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:14.863985 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:14.865094 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:14.866328 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:14.866943 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:14.867122 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:14.869951 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:14.869956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:14.869969 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:14.870016 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:14.871831 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:14.872356 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:14.872366 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:14.872374 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:14.872502 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:14.872819 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:14.872822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:14.872824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:14.873220 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:14.873432 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:14.873753 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 11:44:14.874049 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":146,"threads_started":1}
I20260806 11:44:14.875366 20419 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d5ddebee2a4392af388b67c89b4e55" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34257 } }
I20260806 11:44:14.875627 20419 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:14.875741 20419 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d5ddebee2a4392af388b67c89b4e55, State: Initialized, Role: FOLLOWER
I20260806 11:44:14.876209 20419 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d5ddebee2a4392af388b67c89b4e55" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34257 } }
I20260806 11:44:14.876395 20419 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:44:14.876509 20419 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:44:14.876627 20419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:14.877688 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:44:14.877776 20419 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d5ddebee2a4392af388b67c89b4e55" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34257 } }
I20260806 11:44:14.878134 20419 leader_election.cc:302] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4d5ddebee2a4392af388b67c89b4e55; no voters:
I20260806 11:44:14.878473 20419 leader_election.cc:288] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:44:14.878537 20426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:14.878780 20426 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [term 1 LEADER]: Becoming Leader. State: Replica: a4d5ddebee2a4392af388b67c89b4e55, State: Running, Role: LEADER
I20260806 11:44:14.879091 20426 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d5ddebee2a4392af388b67c89b4e55" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34257 } }
I20260806 11:44:14.879774 20419 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:14.880380 20426 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d5ddebee2a4392af388b67c89b4e55. Latest consensus state: current_term: 1 leader_uuid: "a4d5ddebee2a4392af388b67c89b4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d5ddebee2a4392af388b67c89b4e55" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34257 } } }
I20260806 11:44:14.880506 20426 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:14.880726 20426 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4d5ddebee2a4392af388b67c89b4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d5ddebee2a4392af388b67c89b4e55" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34257 } } }
I20260806 11:44:14.880831 20426 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55 [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:14.882179 20437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:44:14.884739 20437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:44:14.889575 20437 catalog_manager.cc:1347] Generated new cluster ID: 2aa57a751c2241a3bf07f9d1edcbae39
I20260806 11:44:14.889642 20437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:44:14.909348 20437 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:44:14.910339 20437 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:44:14.915612 20437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4d5ddebee2a4392af388b67c89b4e55: Generated new TSK 0
I20260806 11:44:14.916266 20437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:44:14.933449 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 20318
2026-08-06T11:44:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1624 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:44:14 dist-test-slave-45dx krb5kdc[20463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:44:14 dist-test-slave-45dx krb5kdc[20463](info): set up 2 sockets
Aug 06 11:44:14 dist-test-slave-45dx krb5kdc[20463](info): commencing operation
krb5kdc: starting...
W20260806 11:44:16.166669 452 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:44:16 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:44:16.296780 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36125
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:16.410254 20485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:16.410585 20485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:16.410678 20485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:16.414875 20485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:16.414969 20485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:16.415006 20485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:16.415069 20485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:16.415125 20485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:16.420724 20485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:16.421932 20485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:16.423031 20485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:16.432128 20498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:16.432363 20485 server_base.cc:1034] running on GCE node
W20260806 11:44:16.432133 20492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:16.432133 20493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:16.433210 20485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:16.434360 20485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:16.435526 20485 hybrid_clock.cc:648] HybridClock initialized: now 1786016656435507 us; error 30 us; skew 500 ppm
Aug 06 11:44:16 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:16.442549 20485 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:44:16.443585 20485 webserver.cc:460] Webserver started at http://127.0.113.62:39531/ using document root <none> and password file <none>
I20260806 11:44:16.444149 20485 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:16.444213 20485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:16.444468 20485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:16.446588 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "d9fb2316ddfb4e80acf3684873c1f915"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "d62e7e9276fe31d5c7604a4301ab1e9c"
server_key_iv: "50f18670759dbef1b37c966c3b9964da"
server_key_version: "encryptionkey@0"
I20260806 11:44:16.447147 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "d9fb2316ddfb4e80acf3684873c1f915"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "d62e7e9276fe31d5c7604a4301ab1e9c"
server_key_iv: "50f18670759dbef1b37c966c3b9964da"
server_key_version: "encryptionkey@0"
I20260806 11:44:16.451071 20485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 11:44:16.453778 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:16.455039 20485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:44:16.455140 20485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "d9fb2316ddfb4e80acf3684873c1f915"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "d62e7e9276fe31d5c7604a4301ab1e9c"
server_key_iv: "50f18670759dbef1b37c966c3b9964da"
server_key_version: "encryptionkey@0"
I20260806 11:44:16.455224 20485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:16.485656 20485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:16.489239 20485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:16.489411 20485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:16.496884 20485 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:36125
I20260806 11:44:16.497181 20591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:36125 every 8 connection(s)
I20260806 11:44:16.497902 20485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:44:16.500792 20592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:16.504012 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 20485
I20260806 11:44:16.504127 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:44:16.504449 452 external_mini_cluster.cc:1428] Setting key fc0454b85cd41bffed4a60692b8134b6
I20260806 11:44:16.507484 20592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Bootstrap starting.
I20260806 11:44:16.510138 20592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:16.511125 20592 log.cc:824] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:16.513674 20592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: No bootstrap required, opened a new log
Aug 06 11:44:16 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:16.516839 20592 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:16.517112 20592 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:16.517197 20592 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fb2316ddfb4e80acf3684873c1f915, State: Initialized, Role: FOLLOWER
I20260806 11:44:16.517661 20592 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:16.517823 20592 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:44:16.517916 20592 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:44:16.518028 20592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:16.519021 20592 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:16.519383 20592 leader_election.cc:302] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9fb2316ddfb4e80acf3684873c1f915; no voters:
I20260806 11:44:16.519774 20592 leader_election.cc:288] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:44:16.519877 20597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:16.520128 20597 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 1 LEADER]: Becoming Leader. State: Replica: d9fb2316ddfb4e80acf3684873c1f915, State: Running, Role: LEADER
I20260806 11:44:16.520483 20597 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:16.520888 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:16.506129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.0.113.62:36125)
0806 11:44:16.506676 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:16.506728 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:16.506778 (+ 50us) server_negotiation.cc:407] Connection header received
0806 11:44:16.507517 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:16.507535 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:16.507921 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:16.508205 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:16.509214 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:16.510212 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:16.511426 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:16.511981 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:16.512142 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:16.514957 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:16.514963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:16.514975 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:16.515004 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:16.517532 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:16.518028 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:16.518034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:16.518040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:16.518203 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:16.518518 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:16.518522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:16.518524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:16.518956 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:16.519180 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:16.519453 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 11:44:16.519831 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":150,"threads_started":1}
I20260806 11:44:16.521226 20592 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:16.521744 20597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9fb2316ddfb4e80acf3684873c1f915. Latest consensus state: current_term: 1 leader_uuid: "d9fb2316ddfb4e80acf3684873c1f915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } } }
I20260806 11:44:16.521879 20597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:16.522374 20599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9fb2316ddfb4e80acf3684873c1f915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } } }
I20260806 11:44:16.522502 20599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:16.523029 20606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:44:16.525996 20606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:44:16.531239 20606 catalog_manager.cc:1347] Generated new cluster ID: 1f70bfb759dc401ea7e96ae02461cc87
I20260806 11:44:16.531309 20606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:44:16.546792 20606 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:44:16.548081 20606 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:44:16.554207 20606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Generated new TSK 0
I20260806 11:44:16.555011 20606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:44:16.587702 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36125
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:16.703047 20635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:16.703367 20635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:16.703460 20635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:16.707563 20635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:16.707661 20635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:16.707751 20635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:44:16.713162 20635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:16.714267 20635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:16.715335 20635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:16.725208 20645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:16.725208 20641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:16.725210 20643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:16.725561 20635 server_base.cc:1034] running on GCE node
I20260806 11:44:16.725975 20635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:16.726550 20635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:16.727743 20635 hybrid_clock.cc:648] HybridClock initialized: now 1786016656727727 us; error 28 us; skew 500 ppm
Aug 06 11:44:16 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:16.730638 20635 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:44:16.731488 20635 webserver.cc:460] Webserver started at http://127.0.113.1:33971/ using document root <none> and password file <none>
I20260806 11:44:16.732050 20635 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:16.732144 20635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:16.732450 20635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:16.734463 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "30ecea34079842c8a5db2dc5061f7f98"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a9e68cfdd52707d40c729aa43b260d2"
server_key_iv: "7a2ba2fd30710d0e68ad1910b5b56c08"
server_key_version: "encryptionkey@0"
I20260806 11:44:16.735052 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "30ecea34079842c8a5db2dc5061f7f98"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a9e68cfdd52707d40c729aa43b260d2"
server_key_iv: "7a2ba2fd30710d0e68ad1910b5b56c08"
server_key_version: "encryptionkey@0"
I20260806 11:44:16.738776 20635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:44:16.741279 20653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:16.742537 20635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:44:16.742673 20635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "30ecea34079842c8a5db2dc5061f7f98"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a9e68cfdd52707d40c729aa43b260d2"
server_key_iv: "7a2ba2fd30710d0e68ad1910b5b56c08"
server_key_version: "encryptionkey@0"
I20260806 11:44:16.742789 20635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:16.768574 20635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:16.771996 20635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:16.772210 20635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:16.772858 20635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:16.773708 20635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:16.773782 20635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:16.773856 20635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:16.773908 20635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:16.782792 20635 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:40557
I20260806 11:44:16.782817 20840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:40557 every 8 connection(s)
I20260806 11:44:16.783828 20635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:44:16.785106 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 20635
I20260806 11:44:16.785259 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:44:16.785561 452 external_mini_cluster.cc:1428] Setting key 50b442e5f7785a576aed038069984af8
Aug 06 11:44:16 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:44:16.798803 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:16.786279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:60811 (local address 127.0.113.62:36125)
0806 11:44:16.786431 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:16.786436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:16.787348 (+ 912us) server_negotiation.cc:407] Connection header received
0806 11:44:16.788762 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:16.788769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:16.788832 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:16.788938 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:16.790089 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:16.790627 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:16.791856 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:16.792094 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:16.792197 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:16.794912 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:16.794915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:16.794918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:16.794952 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:16.796566 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:16.797130 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:16.797133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:16.797135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:16.797221 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:16.797588 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:16.797592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:16.797594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:16.797778 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:16.797888 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:16.798493 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 11:44:16.798623 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 11:44:16.799638 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:16.786701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:60811)
0806 11:44:16.787173 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:16.787237 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:16.788562 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:16.789098 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:16.789104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:16.789467 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:16.789940 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:16.789954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:16.790756 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:16.790760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:16.791708 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:16.791731 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:16.792251 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:16.792255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:16.792444 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:16.793042 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:16.793058 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:16.794748 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:16.796685 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:16.796692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:16.796706 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:16.797021 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:16.797327 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:16.797331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:16.797333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:16.797482 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:16.797915 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:16.797922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:16.798200 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 11:44:16.798440 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 11:44:16.798779 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":145,"threads_started":1}
I20260806 11:44:16.801121 20841 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36125
I20260806 11:44:16.801401 20841 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:16.801931 20841 heartbeater.cc:511] Master 127.0.113.62:36125 requested a full tablet report, sending...
I20260806 11:44:16.803628 20536 ts_manager.cc:194] Registered new tserver with Master: 30ecea34079842c8a5db2dc5061f7f98 (127.0.113.1:40557)
I20260806 11:44:16.805461 20536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:60811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:44:16.817322 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36125
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:16.932797 20849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:16.933102 20849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:16.933185 20849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:16.937520 20849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:16.937595 20849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:16.937657 20849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:44:16.943257 20849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:16.944551 20849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:16.945640 20849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:16.955992 20856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:16.956007 20860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:16.956059 20849 server_base.cc:1034] running on GCE node
W20260806 11:44:16.956004 20858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:16.956825 20849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:16.957491 20849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:16.958691 20849 hybrid_clock.cc:648] HybridClock initialized: now 1786016656958669 us; error 31 us; skew 500 ppm
Aug 06 11:44:16 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:16.961714 20849 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:44:16.962622 20849 webserver.cc:460] Webserver started at http://127.0.113.2:43883/ using document root <none> and password file <none>
I20260806 11:44:16.963207 20849 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:16.963271 20849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:16.963522 20849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:16.965745 20849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "ce65e14303804dc2a1d63f61358d7c6e"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a0828fec71cef2b80688bf7f34d1fa8"
server_key_iv: "71e5bac2b2488241d3a2d3dba2426b53"
server_key_version: "encryptionkey@0"
I20260806 11:44:16.966355 20849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "ce65e14303804dc2a1d63f61358d7c6e"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a0828fec71cef2b80688bf7f34d1fa8"
server_key_iv: "71e5bac2b2488241d3a2d3dba2426b53"
server_key_version: "encryptionkey@0"
I20260806 11:44:16.970252 20849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 11:44:16.972774 20869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:16.974026 20849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:44:16.974155 20849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "ce65e14303804dc2a1d63f61358d7c6e"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a0828fec71cef2b80688bf7f34d1fa8"
server_key_iv: "71e5bac2b2488241d3a2d3dba2426b53"
server_key_version: "encryptionkey@0"
I20260806 11:44:16.974270 20849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:16.991561 20849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:16.994889 20849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:16.995110 20849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:16.995714 20849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:16.996620 20849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:16.996694 20849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:16.996770 20849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:16.996816 20849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:17.005777 20849 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:34037
I20260806 11:44:17.005812 21048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:34037 every 8 connection(s)
I20260806 11:44:17.006706 20849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:44:17.015293 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 20849
I20260806 11:44:17.015456 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:44:17.015793 452 external_mini_cluster.cc:1428] Setting key 502202d4ed36c501aa42a1ddd9673582
Aug 06 11:44:17 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:17.021055 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.008638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:54823 (local address 127.0.113.62:36125)
0806 11:44:17.008806 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.008811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.009654 (+ 843us) server_negotiation.cc:407] Connection header received
0806 11:44:17.010915 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.010918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.010972 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.011049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:17.012202 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.012818 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.013990 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.014217 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.014333 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.017280 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:17.017286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:17.017293 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:17.017323 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:17.018859 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.019378 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.019383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.019385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.019475 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.019856 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.019859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.019861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.020053 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:17.020163 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.020736 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.020881 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 11:44:17.021971 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.009052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:54823)
0806 11:44:17.009499 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.009543 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:17.010720 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:17.011211 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:17.011217 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:17.011591 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:17.012056 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.012069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.012962 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.012966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.013844 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.013854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.014379 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.014383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.014558 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.015126 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:17.015147 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:17.017106 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:17.018988 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:17.018994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:17.019008 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:17.019270 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:17.019579 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:17.019582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:17.019584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:17.019750 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:17.020161 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:17.020168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:17.020498 (+ 330us) client_negotiation.cc:769] Sending connection context
0806 11:44:17.020692 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 11:44:17.021028 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":115,"threads_started":1}
I20260806 11:44:17.023259 21052 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36125
I20260806 11:44:17.023538 21052 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:17.024063 21052 heartbeater.cc:511] Master 127.0.113.62:36125 requested a full tablet report, sending...
I20260806 11:44:17.025242 20536 ts_manager.cc:194] Registered new tserver with Master: ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037)
I20260806 11:44:17.026055 20536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:54823
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:44:17.047374 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36125
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:17.162492 21059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:17.162739 21059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:17.162809 21059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:17.166740 21059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:17.166801 21059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:17.166906 21059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:44:17.171979 21059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:17.173070 21059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:17.174117 21059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:17.184031 21066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:17.184194 21059 server_base.cc:1034] running on GCE node
W20260806 11:44:17.184083 21065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:17.184031 21068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:17.184814 21059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:17.185479 21059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:17.186640 21059 hybrid_clock.cc:648] HybridClock initialized: now 1786016657186627 us; error 28 us; skew 500 ppm
Aug 06 11:44:17 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:17.189491 21059 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:44:17.190297 21059 webserver.cc:460] Webserver started at http://127.0.113.3:39267/ using document root <none> and password file <none>
I20260806 11:44:17.190806 21059 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:17.190865 21059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:17.191035 21059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:17.193111 21059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "3af94141ac0f483c83c96d82f7fbf2b9"
format_stamp: "Formatted at 2026-08-06 11:44:17 on dist-test-slave-45dx"
server_key: "0e0a7040e20f99615f54fb3e9a7e66a6"
server_key_iv: "d8e2e55356a7d6bf12a2d61469785d92"
server_key_version: "encryptionkey@0"
I20260806 11:44:17.193603 21059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "3af94141ac0f483c83c96d82f7fbf2b9"
format_stamp: "Formatted at 2026-08-06 11:44:17 on dist-test-slave-45dx"
server_key: "0e0a7040e20f99615f54fb3e9a7e66a6"
server_key_iv: "d8e2e55356a7d6bf12a2d61469785d92"
server_key_version: "encryptionkey@0"
I20260806 11:44:17.197196 21059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:44:17.199560 21085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:17.200841 21059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:44:17.200965 21059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "3af94141ac0f483c83c96d82f7fbf2b9"
format_stamp: "Formatted at 2026-08-06 11:44:17 on dist-test-slave-45dx"
server_key: "0e0a7040e20f99615f54fb3e9a7e66a6"
server_key_iv: "d8e2e55356a7d6bf12a2d61469785d92"
server_key_version: "encryptionkey@0"
I20260806 11:44:17.201073 21059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:17.213157 21059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:17.216574 21059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:17.216814 21059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:17.217411 21059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:17.218266 21059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:17.218313 21059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:17.218385 21059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:17.218421 21059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:17.229410 21059 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:44647
I20260806 11:44:17.229436 21251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:44647 every 8 connection(s)
I20260806 11:44:17.230402 21059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:44:17.234723 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 21059
I20260806 11:44:17.234874 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:44:17.235227 452 external_mini_cluster.cc:1428] Setting key 24205a6ac825b34b757ed114b0544c8c
Aug 06 11:44:17 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:17.244740 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.232443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:44463 (local address 127.0.113.62:36125)
0806 11:44:17.232589 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.232593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.233478 (+ 885us) server_negotiation.cc:407] Connection header received
0806 11:44:17.234862 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.234865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.234925 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.235021 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:17.236640 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.237392 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.238517 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.238723 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.238798 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.241309 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:17.241313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:17.241315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:17.241339 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:17.242819 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.243299 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.243302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.243303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.243376 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.243687 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.243689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.243690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.243852 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:17.243924 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.244436 (+ 512us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.244580 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:44:17.245661 21254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.232861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:44463)
0806 11:44:17.233318 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.233361 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:17.234647 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:17.235231 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:17.235241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:17.235733 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:17.236436 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.236457 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.237542 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.237545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.238407 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.238416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.238849 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.238851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.238985 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.239594 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:17.239610 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:17.241157 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:17.242927 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:17.242932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:17.242944 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:17.243202 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:17.243452 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:17.243454 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:17.243456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:17.243600 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:17.243936 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:17.243940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:17.244162 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 11:44:17.244369 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 11:44:17.244743 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260806 11:44:17.246848 21252 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36125
I20260806 11:44:17.247100 21252 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:17.247591 21252 heartbeater.cc:511] Master 127.0.113.62:36125 requested a full tablet report, sending...
I20260806 11:44:17.248649 20536 ts_manager.cc:194] Registered new tserver with Master: 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647)
I20260806 11:44:17.249217 20536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:44463
I20260806 11:44:17.249269 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:44:17.258602 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.251411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.0.113.62:36125)
0806 11:44:17.251546 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.251550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.251702 (+ 152us) server_negotiation.cc:407] Connection header received
0806 11:44:17.251782 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.251785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.251819 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.251882 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:17.252606 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.253092 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.253876 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.254078 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.254149 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.255392 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:17.255395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:17.255397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:17.255419 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:17.256848 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.257224 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.257227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.257228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.257300 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.257607 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.257611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.257613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.257907 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:17.258036 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.258267 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.258415 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 11:44:17.261874 20536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:44:17.264904 20536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:44:17.279048 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.273579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.0.113.2:34037)
0806 11:44:17.274030 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.274044 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.274068 (+ 24us) server_negotiation.cc:407] Connection header received
0806 11:44:17.274148 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.274154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.274349 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.274518 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:17.274986 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.276167 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.277609 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.278233 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.278355 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.278504 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.278800 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.278893 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":124,"threads_started":1}
I20260806 11:44:17.279027 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.272902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:40557 (local address 127.0.0.1:37736)
0806 11:44:17.273291 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.273344 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:17.273492 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:17.274526 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:17.274534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:17.274563 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:17.274714 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.274722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.275901 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.275907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.277268 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.277278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.278454 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.278458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.278573 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.278599 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:44:17.278703 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 11:44:17.278834 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":48,"thread_start_us":117,"threads_started":1}
I20260806 11:44:17.278995 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.273202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:34037 (local address 127.0.0.1:57426)
0806 11:44:17.273748 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.273762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:17.273842 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:17.274713 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:17.274715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:17.274732 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:17.274840 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.274846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.276337 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.276341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.277456 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.277464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.278368 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.278371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.278494 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.278597 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 11:44:17.278758 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 11:44:17.278844 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":53,"thread_start_us":93,"threads_started":1}
I20260806 11:44:17.279526 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.273069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.0.113.1:40557)
0806 11:44:17.274060 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.274075 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.274094 (+ 19us) server_negotiation.cc:407] Connection header received
0806 11:44:17.274157 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.274161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.274349 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.274517 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:17.274844 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.275784 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.277399 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.278316 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.279099 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.279252 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.279303 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.279383 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":99,"threads_started":1}
I20260806 11:44:17.280081 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.273675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:44647 (local address 127.0.0.1:46022)
0806 11:44:17.274526 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.274540 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:17.274627 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:17.275338 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:17.275340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:17.275359 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:17.275439 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.275445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.276791 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.276794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.277902 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.277913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.279665 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.279669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.279774 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.279798 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 11:44:17.279875 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 11:44:17.279943 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260806 11:44:17.280542 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.273967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.113.3:44647)
0806 11:44:17.274842 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.274855 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.274880 (+ 25us) server_negotiation.cc:407] Connection header received
0806 11:44:17.274980 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.274984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.275157 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.275341 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:17.275554 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.276645 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.278048 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.278870 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.278967 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.279100 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.280258 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.280383 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":166,"threads_started":1}
I20260806 11:44:17.282536 20732 tablet_service.cc:1467] Processing CreateTablet for tablet 649625fba2bb42e1bf9a3eeb26c7f874 (DEFAULT_TABLE table=test-table [id=1ab493ff8c83474d8b8fbea9d5e762ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:17.282547 20948 tablet_service.cc:1467] Processing CreateTablet for tablet 649625fba2bb42e1bf9a3eeb26c7f874 (DEFAULT_TABLE table=test-table [id=1ab493ff8c83474d8b8fbea9d5e762ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:17.283056 21157 tablet_service.cc:1467] Processing CreateTablet for tablet 649625fba2bb42e1bf9a3eeb26c7f874 (DEFAULT_TABLE table=test-table [id=1ab493ff8c83474d8b8fbea9d5e762ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:17.283563 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649625fba2bb42e1bf9a3eeb26c7f874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:17.283604 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649625fba2bb42e1bf9a3eeb26c7f874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:17.284031 21157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649625fba2bb42e1bf9a3eeb26c7f874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:17.289942 21273 tablet_bootstrap.cc:492] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Bootstrap starting.
I20260806 11:44:17.291370 21277 tablet_bootstrap.cc:492] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: Bootstrap starting.
I20260806 11:44:17.291512 21276 tablet_bootstrap.cc:492] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: Bootstrap starting.
I20260806 11:44:17.292367 21273 tablet_bootstrap.cc:654] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:17.293444 21273 log.cc:824] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:17.293479 21277 tablet_bootstrap.cc:654] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:17.293941 21276 tablet_bootstrap.cc:654] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:17.294754 21277 log.cc:824] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:17.294754 21276 log.cc:824] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:17.295919 21273 tablet_bootstrap.cc:492] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: No bootstrap required, opened a new log
I20260806 11:44:17.296144 21273 ts_tablet_manager.cc:1397] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 11:44:17.296593 21276 tablet_bootstrap.cc:492] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: No bootstrap required, opened a new log
I20260806 11:44:17.296778 21276 ts_tablet_manager.cc:1397] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 11:44:17.297075 21277 tablet_bootstrap.cc:492] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: No bootstrap required, opened a new log
I20260806 11:44:17.297253 21277 ts_tablet_manager.cc:1397] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 11:44:17.299268 21273 raft_consensus.cc:348] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.299507 21273 raft_consensus.cc:374] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:17.299561 21273 raft_consensus.cc:729] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ecea34079842c8a5db2dc5061f7f98, State: Initialized, Role: FOLLOWER
I20260806 11:44:17.300060 21273 consensus_queue.cc:260] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.300717 20841 heartbeater.cc:503] Master 127.0.113.62:36125 was elected leader, sending a full tablet report...
I20260806 11:44:17.300624 21276 raft_consensus.cc:348] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.300884 21276 raft_consensus.cc:374] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:17.300941 21276 raft_consensus.cc:729] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce65e14303804dc2a1d63f61358d7c6e, State: Initialized, Role: FOLLOWER
I20260806 11:44:17.301110 21273 ts_tablet_manager.cc:1428] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260806 11:44:17.301373 21276 consensus_queue.cc:260] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.301636 21277 raft_consensus.cc:348] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.301937 21277 raft_consensus.cc:374] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:17.302016 21277 raft_consensus.cc:729] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af94141ac0f483c83c96d82f7fbf2b9, State: Initialized, Role: FOLLOWER
I20260806 11:44:17.302279 21276 ts_tablet_manager.cc:1428] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 11:44:17.302452 21277 consensus_queue.cc:260] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.302641 21052 heartbeater.cc:503] Master 127.0.113.62:36125 was elected leader, sending a full tablet report...
I20260806 11:44:17.303133 21252 heartbeater.cc:503] Master 127.0.113.62:36125 was elected leader, sending a full tablet report...
I20260806 11:44:17.303376 21277 ts_tablet_manager.cc:1428] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 11:44:17.334314 21282 raft_consensus.cc:482] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:17.334586 21282 raft_consensus.cc:504] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.335758 21282 leader_election.cc:288] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647), ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037)
I20260806 11:44:17.339974 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.336156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:43975 (local address 127.0.113.3:44647)
0806 11:44:17.336305 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.336309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.336668 (+ 359us) server_negotiation.cc:407] Connection header received
0806 11:44:17.336789 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.336792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.336864 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.336942 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:17.337297 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.337776 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.338951 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.339353 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.339471 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.339538 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.339742 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.339826 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 11:44:17.339994 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.336140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:43543 (local address 127.0.113.2:34037)
0806 11:44:17.336290 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.336295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.336594 (+ 299us) server_negotiation.cc:407] Connection header received
0806 11:44:17.336710 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.336716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.336781 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.336862 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:17.337226 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.337762 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.338934 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.339356 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.339481 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.339542 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.339774 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.339852 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:44:17.340023 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.336004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:44647 (local address 127.0.113.1:43975)
0806 11:44:17.336526 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.336561 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:17.336688 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:17.336980 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:17.336983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:17.337000 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:17.337177 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.337184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.337921 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.337924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.338786 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.338796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.339489 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.339493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.339599 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.339625 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:44:17.339696 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 11:44:17.339847 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":287,"thread_start_us":133,"threads_started":1}
I20260806 11:44:17.340044 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.336003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:34037 (local address 127.0.113.1:43543)
0806 11:44:17.336456 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.336472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:17.336595 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:17.336886 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:17.336892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:17.336915 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:17.337087 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.337094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.337882 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.337886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.338777 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.338787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.339500 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.339504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.339622 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.339640 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 11:44:17.339734 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 11:44:17.339874 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":51,"thread_start_us":140,"threads_started":1}
I20260806 11:44:17.340739 21186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649625fba2bb42e1bf9a3eeb26c7f874" candidate_uuid: "30ecea34079842c8a5db2dc5061f7f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" is_pre_election: true
I20260806 11:44:17.340843 20980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649625fba2bb42e1bf9a3eeb26c7f874" candidate_uuid: "30ecea34079842c8a5db2dc5061f7f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce65e14303804dc2a1d63f61358d7c6e" is_pre_election: true
I20260806 11:44:17.341115 21186 raft_consensus.cc:2454] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ecea34079842c8a5db2dc5061f7f98 in term 0.
I20260806 11:44:17.341177 20980 raft_consensus.cc:2454] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ecea34079842c8a5db2dc5061f7f98 in term 0.
I20260806 11:44:17.341678 20655 leader_election.cc:302] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30ecea34079842c8a5db2dc5061f7f98, 3af94141ac0f483c83c96d82f7fbf2b9, ce65e14303804dc2a1d63f61358d7c6e; no voters:
I20260806 11:44:17.342020 21282 raft_consensus.cc:2781] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:44:17.342116 21282 raft_consensus.cc:482] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:44:17.342162 21282 raft_consensus.cc:3037] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:17.343223 21282 raft_consensus.cc:504] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.343603 21282 leader_election.cc:288] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 1 election: Requested vote from peers 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647), ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037)
I20260806 11:44:17.343995 21186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649625fba2bb42e1bf9a3eeb26c7f874" candidate_uuid: "30ecea34079842c8a5db2dc5061f7f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af94141ac0f483c83c96d82f7fbf2b9"
I20260806 11:44:17.344036 20980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649625fba2bb42e1bf9a3eeb26c7f874" candidate_uuid: "30ecea34079842c8a5db2dc5061f7f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce65e14303804dc2a1d63f61358d7c6e"
I20260806 11:44:17.344138 21186 raft_consensus.cc:3037] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:17.344152 20980 raft_consensus.cc:3037] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:17.345516 21186 raft_consensus.cc:2454] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ecea34079842c8a5db2dc5061f7f98 in term 1.
I20260806 11:44:17.345516 20980 raft_consensus.cc:2454] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ecea34079842c8a5db2dc5061f7f98 in term 1.
I20260806 11:44:17.345813 20654 leader_election.cc:302] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ecea34079842c8a5db2dc5061f7f98, ce65e14303804dc2a1d63f61358d7c6e; no voters:
I20260806 11:44:17.346048 21282 raft_consensus.cc:2781] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:17.346295 21282 raft_consensus.cc:686] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 1 LEADER]: Becoming Leader. State: Replica: 30ecea34079842c8a5db2dc5061f7f98, State: Running, Role: LEADER
I20260806 11:44:17.346630 21282 consensus_queue.cc:237] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.350010 20532 catalog_manager.cc:5495] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ecea34079842c8a5db2dc5061f7f98 (127.0.113.1). New cstate: current_term: 1 leader_uuid: "30ecea34079842c8a5db2dc5061f7f98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260806 11:44:17.376581 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.373045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.0.113.1:40557)
0806 11:44:17.373187 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.373193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.373230 (+ 37us) server_negotiation.cc:407] Connection header received
0806 11:44:17.373313 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.373316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.373371 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.373480 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:44:17.373735 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.374230 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.375082 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.375289 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.375405 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.375749 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:44:17.376141 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:44:17.376264 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.376387 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.376442 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:44:17.382397 21186 raft_consensus.cc:1261] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [term 1 FOLLOWER]: Refusing update from remote peer 30ecea34079842c8a5db2dc5061f7f98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:17.382393 20980 raft_consensus.cc:1261] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [term 1 FOLLOWER]: Refusing update from remote peer 30ecea34079842c8a5db2dc5061f7f98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:17.383118 21289 consensus_queue.cc:1035] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:17.383286 21282 consensus_queue.cc:1035] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:17.389734 21294 mvcc.cc:205] Tried to move back new op lower bound from 7315524228631289856 to 7315524228494860288. Current Snapshot: MvccSnapshot[applied={T|T < 7315524228631289856}]
I20260806 11:44:17.394975 21296 mvcc.cc:205] Tried to move back new op lower bound from 7315524228631289856 to 7315524228494860288. Current Snapshot: MvccSnapshot[applied={T|T < 7315524228631289856}]
I20260806 11:44:17.395172 21295 mvcc.cc:205] Tried to move back new op lower bound from 7315524228631289856 to 7315524228494860288. Current Snapshot: MvccSnapshot[applied={T|T < 7315524228631289856 or (T in {7315524228631289856})}]
I20260806 11:44:17.399549 20536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:17.399729 20536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:17.402211 20536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Sending DeleteTablet for 3 replicas of tablet 649625fba2bb42e1bf9a3eeb26c7f874
I20260806 11:44:17.403100 20948 tablet_service.cc:1514] Processing DeleteTablet for tablet 649625fba2bb42e1bf9a3eeb26c7f874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:17 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:57426
I20260806 11:44:17.403252 21157 tablet_service.cc:1514] Processing DeleteTablet for tablet 649625fba2bb42e1bf9a3eeb26c7f874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:17 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:46022
I20260806 11:44:17.403254 20732 tablet_service.cc:1514] Processing DeleteTablet for tablet 649625fba2bb42e1bf9a3eeb26c7f874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:17 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:37736
I20260806 11:44:17.403581 21307 tablet_replica.cc:330] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: stopping tablet replica
I20260806 11:44:17.403874 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 20635
I20260806 11:44:17.403877 21307 raft_consensus.cc:2229] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:17.410032 21307 raft_consensus.cc:2258] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:17.410140 21308 tablet_replica.cc:330] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: stopping tablet replica
I20260806 11:44:17.410449 21308 raft_consensus.cc:2229] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:17.410734 21308 raft_consensus.cc:2258] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:17.411909 21307 ts_tablet_manager.cc:1905] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:17.412988 21308 ts_tablet_manager.cc:1905] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:17.415455 21308 ts_tablet_manager.cc:1918] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:17.415580 21308 log.cc:1196] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/649625fba2bb42e1bf9a3eeb26c7f874
I20260806 11:44:17.415634 21307 ts_tablet_manager.cc:1918] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:17.415747 21307 log.cc:1196] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/649625fba2bb42e1bf9a3eeb26c7f874
I20260806 11:44:17.415954 21308 ts_tablet_manager.cc:1939] T 649625fba2bb42e1bf9a3eeb26c7f874 P 3af94141ac0f483c83c96d82f7fbf2b9: Deleting consensus metadata
I20260806 11:44:17.416096 21307 ts_tablet_manager.cc:1939] T 649625fba2bb42e1bf9a3eeb26c7f874 P ce65e14303804dc2a1d63f61358d7c6e: Deleting consensus metadata
I20260806 11:44:17.417129 20510 catalog_manager.cc:4841] TS 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647): tablet 649625fba2bb42e1bf9a3eeb26c7f874 (table test-table [id=1ab493ff8c83474d8b8fbea9d5e762ed]) successfully deleted
I20260806 11:44:17.417268 20509 catalog_manager.cc:4841] TS ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037): tablet 649625fba2bb42e1bf9a3eeb26c7f874 (table test-table [id=1ab493ff8c83474d8b8fbea9d5e762ed]) successfully deleted
W20260806 11:44:17.420538 20510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:40557 (error 108)
I20260806 11:44:17.420789 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 20849
I20260806 11:44:17.424767 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.424409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:40557 (local address 127.0.0.1:37748)
0806 11:44:17.424600 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.424672 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260806 11:44:17.424970 20510 catalog_manager.cc:4584] TS 30ecea34079842c8a5db2dc5061f7f98 (127.0.113.1:40557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 649625fba2bb42e1bf9a3eeb26c7f874: Network error: Client connection negotiation failed: client connection to 127.0.113.1:40557: connect: Connection refused (error 111)
I20260806 11:44:17.431499 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 21059
I20260806 11:44:17.446534 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 20485
I20260806 11:44:17.455237 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36125
--webserver_interface=127.0.113.62
--webserver_port=39531
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:17.562441 21311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:17.562757 21311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:17.562832 21311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:17.566710 21311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:17.566790 21311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:17.566826 21311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:17.566882 21311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:17.566918 21311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:17.571910 21311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:17.572928 21311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:17.573891 21311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:17.587188 21328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:17.587189 21326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:17.587188 21311 server_base.cc:1034] running on GCE node
W20260806 11:44:17.587209 21323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:17.588030 21311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:17.589177 21311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:17.590339 21311 hybrid_clock.cc:648] HybridClock initialized: now 1786016657590323 us; error 26 us; skew 500 ppm
Aug 06 11:44:17 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:17.593313 21311 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:44:17.594170 21311 webserver.cc:460] Webserver started at http://127.0.113.62:39531/ using document root <none> and password file <none>
I20260806 11:44:17.594756 21311 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:17.594870 21311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:17.598891 21311 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:44:17.601125 21337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:17.602344 21311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:44:17.602510 21311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "d9fb2316ddfb4e80acf3684873c1f915"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "d62e7e9276fe31d5c7604a4301ab1e9c"
server_key_iv: "50f18670759dbef1b37c966c3b9964da"
server_key_version: "encryptionkey@0"
I20260806 11:44:17.603127 21311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:17.624966 21311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:17.628566 21311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:17.628782 21311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:17.636051 21311 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:36125
I20260806 11:44:17.636382 21418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:36125 every 8 connection(s)
I20260806 11:44:17.637236 21311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:44:17.641897 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 21311
I20260806 11:44:17.641978 21419 sys_catalog.cc:263] Verifying existing consensus state
I20260806 11:44:17.642377 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:40557
--local_ip_for_outbound_sockets=127.0.113.1
--tserver_master_addrs=127.0.113.62:36125
--webserver_port=33971
--webserver_interface=127.0.113.1
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:44:17.643687 21419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Bootstrap starting.
I20260806 11:44:17.667042 21419 log.cc:824] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:17.677820 21419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:44:17.678226 21419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Bootstrap complete.
I20260806 11:44:17.683701 21419 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:17.684188 21419 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fb2316ddfb4e80acf3684873c1f915, State: Initialized, Role: FOLLOWER
I20260806 11:44:17.684729 21419 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:17.684876 21419 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:44:17.684958 21419 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:44:17.685051 21419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 1 FOLLOWER]: Advancing to term 2
I20260806 11:44:17.687604 21419 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:17.687974 21419 leader_election.cc:302] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9fb2316ddfb4e80acf3684873c1f915; no voters:
I20260806 11:44:17.688335 21419 leader_election.cc:288] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 11:44:17.688457 21424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 11:44:17.688730 21424 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 2 LEADER]: Becoming Leader. State: Replica: d9fb2316ddfb4e80acf3684873c1f915, State: Running, Role: LEADER
I20260806 11:44:17.689039 21424 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:17.689548 21419 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:17.689916 21424 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9fb2316ddfb4e80acf3684873c1f915. Latest consensus state: current_term: 2 leader_uuid: "d9fb2316ddfb4e80acf3684873c1f915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } } }
I20260806 11:44:17.690069 21424 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:17.690348 21426 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d9fb2316ddfb4e80acf3684873c1f915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } } }
I20260806 11:44:17.690447 21426 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:17.691794 21429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:44:17.696394 21429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:44:17.696990 21429 catalog_manager.cc:1259] Loaded cluster ID: 1f70bfb759dc401ea7e96ae02461cc87
I20260806 11:44:17.697053 21429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:44:17.698416 21429 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 11:44:17.703023 21429 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 11:44:17.707968 21429 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:44:17.708624 21429 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Loaded TSK: 0
I20260806 11:44:17.709719 21429 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 11:44:17.771656 21422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:17.771889 21422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:17.771930 21422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:17.775794 21422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:17.775843 21422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:17.775897 21422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:44:17.780869 21422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:40557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=33971
--tserver_master_addrs=127.0.113.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:17.781857 21422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:17.782809 21422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:17.792964 21458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:17.792953 21459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:17.793244 21462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:17.793562 21422 server_base.cc:1034] running on GCE node
I20260806 11:44:17.793934 21422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:17.794597 21422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:17.795776 21422 hybrid_clock.cc:648] HybridClock initialized: now 1786016657795756 us; error 36 us; skew 500 ppm
Aug 06 11:44:17 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:17.798528 21422 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:44:17.799444 21422 webserver.cc:460] Webserver started at http://127.0.113.1:33971/ using document root <none> and password file <none>
I20260806 11:44:17.800021 21422 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:17.800171 21422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:17.804009 21422 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:44:17.806190 21473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:17.807516 21422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 11:44:17.807638 21422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "30ecea34079842c8a5db2dc5061f7f98"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a9e68cfdd52707d40c729aa43b260d2"
server_key_iv: "7a2ba2fd30710d0e68ad1910b5b56c08"
server_key_version: "encryptionkey@0"
I20260806 11:44:17.808158 21422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:17.823647 21422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:17.826429 21422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:17.826623 21422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:17.827251 21422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:17.828400 21481 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 11:44:17.832373 21422 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 11:44:17.832429 21422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260806 11:44:17.832501 21422 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 11:44:17.833987 21422 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 11:44:17.834040 21422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 11:44:17.834278 21481 tablet_bootstrap.cc:492] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Bootstrap starting.
I20260806 11:44:17.845009 21640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:40557 every 8 connection(s)
I20260806 11:44:17.844990 21422 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:40557
I20260806 11:44:17.846186 21422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:44:17.850888 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 21422
I20260806 11:44:17.851413 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:34037
--local_ip_for_outbound_sockets=127.0.113.2
--tserver_master_addrs=127.0.113.62:36125
--webserver_port=43883
--webserver_interface=127.0.113.2
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:44:17.855274 21481 log.cc:824] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Log is configured to *not* fsync() on all Append() calls
Aug 06 11:44:17 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:17.862958 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.848203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:49409 (local address 127.0.113.62:36125)
0806 11:44:17.848559 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.848577 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.848884 (+ 307us) server_negotiation.cc:407] Connection header received
0806 11:44:17.849975 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.849986 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.850309 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.850713 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:17.851956 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.852883 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.854565 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.855196 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.855369 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.858008 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:17.858013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:17.858025 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:17.858051 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:17.859715 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.860291 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.860296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.860339 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.860474 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.860816 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.860821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.860822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.861241 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:17.861377 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.861882 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.862137 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":133,"threads_started":1}
I20260806 11:44:17.862982 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.848215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:49409)
0806 11:44:17.848731 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.848768 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:17.849581 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:17.850748 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:17.850756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:17.851262 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:17.851780 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.851796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.853064 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.853073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.854397 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.854413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.855351 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.855371 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.855612 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.856401 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:17.856416 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:17.857813 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:17.859842 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:17.859847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:17.859860 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:17.860166 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:17.860575 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:17.860579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:17.860581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:17.860720 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:17.861351 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:17.861357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:17.861574 (+ 217us) client_negotiation.cc:769] Sending connection context
0806 11:44:17.861740 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 11:44:17.862087 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":146,"threads_started":1}
I20260806 11:44:17.864886 21481 tablet_bootstrap.cc:492] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:44:17.865341 21481 tablet_bootstrap.cc:492] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Bootstrap complete.
I20260806 11:44:17.865520 21643 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36125
I20260806 11:44:17.865725 21643 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:17.866190 21643 heartbeater.cc:511] Master 127.0.113.62:36125 requested a full tablet report, sending...
I20260806 11:44:17.866778 21481 ts_tablet_manager.cc:1397] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.008s
I20260806 11:44:17.868465 21363 ts_manager.cc:194] Registered new tserver with Master: 30ecea34079842c8a5db2dc5061f7f98 (127.0.113.1:40557)
I20260806 11:44:17.869475 21481 raft_consensus.cc:348] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.869931 21481 raft_consensus.cc:729] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ecea34079842c8a5db2dc5061f7f98, State: Initialized, Role: FOLLOWER
I20260806 11:44:17.870417 21481 consensus_queue.cc:260] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } } peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } }
I20260806 11:44:17.871354 21481 ts_tablet_manager.cc:1428] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260806 11:44:17.871521 21363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:49409
I20260806 11:44:17.874728 21643 heartbeater.cc:503] Master 127.0.113.62:36125 was elected leader, sending a full tablet report...
Aug 06 11:44:17 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:44:17.881769 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.872280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:40557 (local address 127.0.0.1:37754)
0806 11:44:17.872706 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:17.872757 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:17.872906 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:17.873513 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:17.873518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:17.873855 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:17.873949 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.873955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.874893 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.874895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.875671 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:17.875681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.876401 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.876408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:17.876487 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.876970 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:17.876986 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:17.878237 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:17.880107 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:17.880115 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:17.880120 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:17.880424 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:17.880742 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:17.880745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:17.880747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:17.880835 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:17.881311 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:17.881315 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:17.881405 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 11:44:17.881514 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 11:44:17.881647 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
I20260806 11:44:17.881937 21656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:17.872425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.0.113.1:40557)
0806 11:44:17.873089 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:17.873101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:17.873121 (+ 20us) server_negotiation.cc:407] Connection header received
0806 11:44:17.873177 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:17.873182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:17.873359 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:17.873505 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:17.874068 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.874728 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.875807 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:17.876268 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:17.876384 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:17.878360 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:17.878369 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:17.878382 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:17.878411 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:17.879993 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.880535 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.880541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.880543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.880637 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:17.880923 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:17.880925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:17.880927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:17.881198 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:17.881329 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:17.881616 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 11:44:17.881771 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":95,"threads_started":1}
I20260806 11:44:17.883004 21548 tablet_service.cc:1514] Processing DeleteTablet for tablet 649625fba2bb42e1bf9a3eeb26c7f874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:17 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:37754
I20260806 11:44:17.883903 21658 tablet_replica.cc:330] stopping tablet replica
I20260806 11:44:17.884166 21658 raft_consensus.cc:2229] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:17.884464 21658 raft_consensus.cc:2258] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:17.885366 21658 ts_tablet_manager.cc:1905] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:17.888698 21658 ts_tablet_manager.cc:1918] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:17.888808 21658 log.cc:1196] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/649625fba2bb42e1bf9a3eeb26c7f874
I20260806 11:44:17.889076 21658 ts_tablet_manager.cc:1939] T 649625fba2bb42e1bf9a3eeb26c7f874 P 30ecea34079842c8a5db2dc5061f7f98: Deleting consensus metadata
I20260806 11:44:17.890694 21342 catalog_manager.cc:4841] TS 30ecea34079842c8a5db2dc5061f7f98 (127.0.113.1:40557): tablet 649625fba2bb42e1bf9a3eeb26c7f874 (table test-table [id=1ab493ff8c83474d8b8fbea9d5e762ed]) successfully deleted
W20260806 11:44:17.968699 21652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:17.968935 21652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:17.968971 21652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:17.972749 21652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:17.972798 21652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:17.972867 21652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:44:17.977494 21652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:34037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=43883
--tserver_master_addrs=127.0.113.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:17.978299 21652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:17.979246 21652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:17.988622 21671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:17.988554 21667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:17.988570 21669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:17.989265 21652 server_base.cc:1034] running on GCE node
I20260806 11:44:17.989656 21652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:17.990358 21652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:17.991533 21652 hybrid_clock.cc:648] HybridClock initialized: now 1786016657991508 us; error 44 us; skew 500 ppm
Aug 06 11:44:17 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:17.994200 21652 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:44:17.994995 21652 webserver.cc:460] Webserver started at http://127.0.113.2:43883/ using document root <none> and password file <none>
I20260806 11:44:17.995518 21652 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:17.995623 21652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:17.999087 21652 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:44:18.001065 21680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:18.002171 21652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:44:18.002310 21652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "ce65e14303804dc2a1d63f61358d7c6e"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a0828fec71cef2b80688bf7f34d1fa8"
server_key_iv: "71e5bac2b2488241d3a2d3dba2426b53"
server_key_version: "encryptionkey@0"
I20260806 11:44:18.002794 21652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:18.017518 21652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:18.019882 21652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:18.020094 21652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:18.020658 21652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:18.021430 21652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:18.021498 21652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:18.021559 21652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:18.021607 21652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:18.029989 21652 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:34037
I20260806 11:44:18.030040 21842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:34037 every 8 connection(s)
I20260806 11:44:18.030958 21652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
Aug 06 11:44:18 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:18.040194 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 21652
I20260806 11:44:18.040699 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:44647
--local_ip_for_outbound_sockets=127.0.113.3
--tserver_master_addrs=127.0.113.62:36125
--webserver_port=39267
--webserver_interface=127.0.113.3
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:44:18.043977 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.032674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:51623 (local address 127.0.113.62:36125)
0806 11:44:18.032850 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:18.032854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:18.033655 (+ 801us) server_negotiation.cc:407] Connection header received
0806 11:44:18.034881 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:18.034884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:18.034935 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:18.035007 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:18.036091 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.036720 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.037813 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.038034 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.038112 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.040419 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:18.040423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:18.040426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:18.040452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:18.041923 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:18.042437 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:18.042444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:18.042448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:18.042533 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:18.042873 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:18.042877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:18.042879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:18.043078 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:18.043186 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:18.043650 (+ 464us) server_negotiation.cc:299] Negotiation successful
0806 11:44:18.043776 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 11:44:18.044778 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.033079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:51623)
0806 11:44:18.033504 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:18.033543 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:18.034692 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:18.035162 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:18.035168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:18.035504 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:18.035952 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.035964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.036828 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.036831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.037709 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.037719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.038158 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.038161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.038289 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.038817 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:18.038833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:18.040221 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:18.042059 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:18.042064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:18.042077 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:18.042329 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:18.042630 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:18.042633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:18.042635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:18.042772 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:18.043185 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:18.043191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:18.043419 (+ 228us) client_negotiation.cc:769] Sending connection context
0806 11:44:18.043604 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 11:44:18.043921 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":117,"threads_started":1}
I20260806 11:44:18.045907 21843 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36125
I20260806 11:44:18.046175 21843 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:18.046767 21843 heartbeater.cc:511] Master 127.0.113.62:36125 requested a full tablet report, sending...
I20260806 11:44:18.047964 21363 ts_manager.cc:194] Registered new tserver with Master: ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037)
I20260806 11:44:18.048841 21363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:51623
W20260806 11:44:18.170703 21847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:18.170965 21847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:18.171007 21847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:18.175061 21847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:18.175119 21847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:18.175176 21847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:44:18.180483 21847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:44647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=39267
--tserver_master_addrs=127.0.113.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:18.181456 21847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:18.182426 21847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:18.192634 21854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:18.192682 21857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:18.192869 21847 server_base.cc:1034] running on GCE node
W20260806 11:44:18.192636 21855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:18.193425 21847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:18.194115 21847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:18.195309 21847 hybrid_clock.cc:648] HybridClock initialized: now 1786016658195293 us; error 29 us; skew 500 ppm
Aug 06 11:44:18 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:18.198118 21847 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:44:18.199116 21847 webserver.cc:460] Webserver started at http://127.0.113.3:39267/ using document root <none> and password file <none>
I20260806 11:44:18.199699 21847 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:18.199808 21847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:18.203650 21847 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:44:18.205971 21866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:18.207329 21847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 11:44:18.207434 21847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "3af94141ac0f483c83c96d82f7fbf2b9"
format_stamp: "Formatted at 2026-08-06 11:44:17 on dist-test-slave-45dx"
server_key: "0e0a7040e20f99615f54fb3e9a7e66a6"
server_key_iv: "d8e2e55356a7d6bf12a2d61469785d92"
server_key_version: "encryptionkey@0"
I20260806 11:44:18.208004 21847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:18.237253 21847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:18.240254 21847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:18.240476 21847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:18.241007 21847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:18.241765 21847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:18.241827 21847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:18.241868 21847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:18.241886 21847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:18.250656 21847 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:44647
I20260806 11:44:18.250742 22043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:44647 every 8 connection(s)
I20260806 11:44:18.251693 21847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:44:18.259596 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 21847
Aug 06 11:44:18 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:18.266589 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.253420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:53049 (local address 127.0.113.62:36125)
0806 11:44:18.253545 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:18.253548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:18.254252 (+ 704us) server_negotiation.cc:407] Connection header received
0806 11:44:18.255528 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:18.255531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:18.255594 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:18.255697 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:18.256928 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.257444 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.258550 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.258768 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.258877 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.261921 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:18.261926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:18.261931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:18.261965 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:18.263887 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:18.264516 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:18.264554 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:18.264559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:18.264689 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:18.265156 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:18.265161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:18.265165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:18.265498 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:18.265604 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:18.266302 (+ 698us) server_negotiation.cc:299] Negotiation successful
0806 11:44:18.266423 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 11:44:18.267432 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.253691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:53049)
0806 11:44:18.254107 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:18.254147 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:18.255324 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:18.255895 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:18.255901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:18.256239 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:18.256772 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.256794 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.257584 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.257588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.258422 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.258432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.258925 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.258930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.259091 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.259684 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:18.259731 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:18.261689 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:18.264010 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:18.264016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:18.264044 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:18.264394 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:18.264829 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:18.264835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:18.264837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:18.265023 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:18.265667 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:18.265673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:18.265992 (+ 319us) client_negotiation.cc:769] Sending connection context
0806 11:44:18.266223 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 11:44:18.266535 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":113,"threads_started":1}
I20260806 11:44:18.268926 22044 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36125
I20260806 11:44:18.269169 22044 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:18.269750 22044 heartbeater.cc:511] Master 127.0.113.62:36125 requested a full tablet report, sending...
I20260806 11:44:18.271001 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.260380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.0.113.62:36125)
0806 11:44:18.260642 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:18.260645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:18.260763 (+ 118us) server_negotiation.cc:407] Connection header received
0806 11:44:18.260915 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:18.260917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:18.260956 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:18.261062 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:18.262308 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.263212 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.264532 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.264750 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.264861 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.266514 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:18.266518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:18.266520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:18.266545 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:18.268568 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:18.269077 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:18.269080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:18.269082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:18.269170 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:18.269570 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:18.269573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:18.269575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:18.269792 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:18.270085 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:18.270774 (+ 689us) server_negotiation.cc:299] Negotiation successful
0806 11:44:18.270890 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
I20260806 11:44:18.271536 21363 ts_manager.cc:194] Registered new tserver with Master: 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647)
I20260806 11:44:18.272135 21363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:53049
I20260806 11:44:18.273880 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:44:18.277709 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.274482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.0.113.62:36125)
0806 11:44:18.274568 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:18.274571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:18.274783 (+ 212us) server_negotiation.cc:407] Connection header received
0806 11:44:18.274917 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:18.274919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:18.274995 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:18.275073 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:44:18.275249 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.275704 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.276583 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.276817 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.276898 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.277115 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:44:18.277439 (+ 324us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:44:18.277520 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:18.277578 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 11:44:18.277622 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 11:44:18.278264 21362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:44:18.281375 21362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:44:18.288964 21548 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c (DEFAULT_TABLE table=test-table [id=63f63a2369c040d79bf0a48f7c9311df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:18.289575 21548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:18.292264 21481 tablet_bootstrap.cc:492] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Bootstrap starting.
I20260806 11:44:18.293185 21481 tablet_bootstrap.cc:654] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:18.293239 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.287897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:44647 (local address 127.0.0.1:46026)
0806 11:44:18.288430 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:18.288445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:18.288525 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:18.289050 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:18.289052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:18.289070 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:18.289155 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.289161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.290508 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.290512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.291732 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.291743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.292857 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.292861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.292962 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.293008 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 11:44:18.293058 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 11:44:18.293127 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469}
I20260806 11:44:18.293817 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.288041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.0.113.3:44647)
0806 11:44:18.288608 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:18.288620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:18.288641 (+ 21us) server_negotiation.cc:407] Connection header received
0806 11:44:18.288721 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:18.288726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:18.288912 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:18.289070 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:18.289282 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.290343 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.291891 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.292723 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.293409 (+ 686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.293548 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:18.293593 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 11:44:18.293669 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":142,"threads_started":1}
I20260806 11:44:18.294863 21481 tablet_bootstrap.cc:492] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: No bootstrap required, opened a new log
I20260806 11:44:18.294952 21481 ts_tablet_manager.cc:1397] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 11:44:18.295032 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.288210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:34037 (local address 127.0.0.1:57436)
0806 11:44:18.288638 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:18.288651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:18.288732 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:18.290391 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:18.290397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:18.290421 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:18.290526 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.290656 (+ 130us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.292102 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.292105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.293710 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.293720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.294671 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.294675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.294760 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.294781 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 11:44:18.294854 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 11:44:18.294916 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":138,"threads_started":1}
I20260806 11:44:18.295147 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.289297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.0.113.2:34037)
0806 11:44:18.289648 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:18.289660 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:18.289683 (+ 23us) server_negotiation.cc:407] Connection header received
0806 11:44:18.289755 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:18.289759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:18.289922 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:18.290105 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:18.290900 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.291958 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.293847 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.294540 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.294654 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.294786 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:18.294901 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 11:44:18.295005 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":127,"threads_started":1}
I20260806 11:44:18.295420 21481 raft_consensus.cc:348] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:18.295553 21481 raft_consensus.cc:374] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:18.295599 21481 raft_consensus.cc:729] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ecea34079842c8a5db2dc5061f7f98, State: Initialized, Role: FOLLOWER
I20260806 11:44:18.295723 21481 consensus_queue.cc:260] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:18.296060 21481 ts_tablet_manager.cc:1428] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:44:18.296923 21940 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c (DEFAULT_TABLE table=test-table [id=63f63a2369c040d79bf0a48f7c9311df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:18.297873 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:18.297883 21753 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c (DEFAULT_TABLE table=test-table [id=63f63a2369c040d79bf0a48f7c9311df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:18.298921 21753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:18.304291 22067 tablet_bootstrap.cc:492] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: Bootstrap starting.
I20260806 11:44:18.304663 22068 tablet_bootstrap.cc:492] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Bootstrap starting.
I20260806 11:44:18.306558 22067 tablet_bootstrap.cc:654] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:18.307206 22068 tablet_bootstrap.cc:654] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:18.307683 22067 log.cc:824] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:18.307986 22068 log.cc:824] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:18.309731 22068 tablet_bootstrap.cc:492] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: No bootstrap required, opened a new log
I20260806 11:44:18.309731 22067 tablet_bootstrap.cc:492] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: No bootstrap required, opened a new log
I20260806 11:44:18.309926 22068 ts_tablet_manager.cc:1397] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260806 11:44:18.309937 22067 ts_tablet_manager.cc:1397] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 11:44:18.313834 22068 raft_consensus.cc:348] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:18.313834 22067 raft_consensus.cc:348] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:18.314108 22067 raft_consensus.cc:374] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:18.314108 22068 raft_consensus.cc:374] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:18.314168 22068 raft_consensus.cc:729] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce65e14303804dc2a1d63f61358d7c6e, State: Initialized, Role: FOLLOWER
I20260806 11:44:18.314178 22067 raft_consensus.cc:729] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af94141ac0f483c83c96d82f7fbf2b9, State: Initialized, Role: FOLLOWER
I20260806 11:44:18.314646 22068 consensus_queue.cc:260] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:18.314646 22067 consensus_queue.cc:260] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:18.315294 22044 heartbeater.cc:503] Master 127.0.113.62:36125 was elected leader, sending a full tablet report...
I20260806 11:44:18.315550 21843 heartbeater.cc:503] Master 127.0.113.62:36125 was elected leader, sending a full tablet report...
I20260806 11:44:18.315647 22068 ts_tablet_manager.cc:1428] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260806 11:44:18.315761 22067 ts_tablet_manager.cc:1428] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
W20260806 11:44:18.347349 21644 tablet.cc:2369] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:44:18.500260 22072 raft_consensus.cc:482] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:18.500535 22072 raft_consensus.cc:504] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:18.501703 22072 leader_election.cc:288] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037), 30ecea34079842c8a5db2dc5061f7f98 (127.0.113.1:40557)
W20260806 11:44:18.502635 22045 tablet.cc:2369] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:44:18.505705 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.502077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:46583 (local address 127.0.113.1:40557)
0806 11:44:18.502340 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:18.502345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:18.502500 (+ 155us) server_negotiation.cc:407] Connection header received
0806 11:44:18.502596 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:18.502599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:18.502662 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:18.502776 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:18.503038 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.503541 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.504668 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.505111 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.505231 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.505383 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:18.505449 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 11:44:18.505531 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260806 11:44:18.505678 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.501952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:40557 (local address 127.0.113.3:46583)
0806 11:44:18.502359 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:18.502373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:18.502503 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:18.502787 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:18.502790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:18.502805 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:18.502928 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.502933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.503651 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.503654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.504552 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.504562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.505230 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.505234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.505340 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.505362 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 11:44:18.505440 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 11:44:18.505524 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":51,"thread_start_us":118,"threads_started":1}
I20260806 11:44:18.505828 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.501931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:34037 (local address 127.0.113.3:40141)
0806 11:44:18.502120 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:18.502137 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:18.502234 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:18.503156 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:18.503158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:18.503173 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:18.503252 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.503256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.503931 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.503936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.504667 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:18.504676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.505293 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.505296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:18.505455 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.505471 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 11:44:18.505572 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 11:44:18.505649 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":37}
I20260806 11:44:18.506151 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.502031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:40141 (local address 127.0.113.2:34037)
0806 11:44:18.502165 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:18.502169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:18.502186 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:44:18.502374 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:18.502377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:18.502422 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:18.502509 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:18.503372 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.503825 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.504794 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.505180 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.505289 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.505355 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:18.505590 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 11:44:18.505656 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:44:18.506422 21576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcd96109d9e4e32a65ca8523bdd9d5c" candidate_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ecea34079842c8a5db2dc5061f7f98" is_pre_election: true
I20260806 11:44:18.506706 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcd96109d9e4e32a65ca8523bdd9d5c" candidate_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce65e14303804dc2a1d63f61358d7c6e" is_pre_election: true
I20260806 11:44:18.506747 21576 raft_consensus.cc:2454] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af94141ac0f483c83c96d82f7fbf2b9 in term 0.
I20260806 11:44:18.507014 21782 raft_consensus.cc:2454] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af94141ac0f483c83c96d82f7fbf2b9 in term 0.
I20260806 11:44:18.507211 21873 leader_election.cc:302] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ecea34079842c8a5db2dc5061f7f98, 3af94141ac0f483c83c96d82f7fbf2b9; no voters:
I20260806 11:44:18.507498 22072 raft_consensus.cc:2781] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:44:18.507581 22072 raft_consensus.cc:482] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:44:18.507627 22072 raft_consensus.cc:3037] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:18.508638 22072 raft_consensus.cc:504] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:18.508939 22072 leader_election.cc:288] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [CANDIDATE]: Term 1 election: Requested vote from peers ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037), 30ecea34079842c8a5db2dc5061f7f98 (127.0.113.1:40557)
I20260806 11:44:18.509236 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcd96109d9e4e32a65ca8523bdd9d5c" candidate_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce65e14303804dc2a1d63f61358d7c6e"
I20260806 11:44:18.509274 21576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcd96109d9e4e32a65ca8523bdd9d5c" candidate_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ecea34079842c8a5db2dc5061f7f98"
I20260806 11:44:18.509367 21782 raft_consensus.cc:3037] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:18.509423 21576 raft_consensus.cc:3037] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:18.510368 21576 raft_consensus.cc:2454] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af94141ac0f483c83c96d82f7fbf2b9 in term 1.
I20260806 11:44:18.510396 21782 raft_consensus.cc:2454] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af94141ac0f483c83c96d82f7fbf2b9 in term 1.
I20260806 11:44:18.510627 21873 leader_election.cc:302] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ecea34079842c8a5db2dc5061f7f98, 3af94141ac0f483c83c96d82f7fbf2b9; no voters:
I20260806 11:44:18.510809 22072 raft_consensus.cc:2781] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:18.511041 22072 raft_consensus.cc:686] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 1 LEADER]: Becoming Leader. State: Replica: 3af94141ac0f483c83c96d82f7fbf2b9, State: Running, Role: LEADER
I20260806 11:44:18.511380 22072 consensus_queue.cc:237] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:18.514432 21362 catalog_manager.cc:5495] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3). New cstate: current_term: 1 leader_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } health_report { overall_health: UNKNOWN } } }
W20260806 11:44:18.532141 21844 tablet.cc:2369] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:44:18.536896 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.533834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.0.113.3:44647)
0806 11:44:18.533957 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:18.533961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:18.533973 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:44:18.534073 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:18.534075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:18.534124 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:18.534205 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:44:18.534398 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.534855 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.535575 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:18.535764 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:18.535835 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:18.536078 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:44:18.536515 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:44:18.536603 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:18.536655 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 11:44:18.536701 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:44:18.542380 21782 raft_consensus.cc:1261] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [term 1 FOLLOWER]: Refusing update from remote peer 3af94141ac0f483c83c96d82f7fbf2b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:18.542377 21576 raft_consensus.cc:1261] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Refusing update from remote peer 3af94141ac0f483c83c96d82f7fbf2b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:18.543066 22072 consensus_queue.cc:1035] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:18.543264 22076 consensus_queue.cc:1035] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:18.550583 22081 mvcc.cc:205] Tried to move back new op lower bound from 7315524233382809600 to 7315524233265143808. Current Snapshot: MvccSnapshot[applied={T|T < 7315524233382809600}]
I20260806 11:44:18.553081 22085 mvcc.cc:205] Tried to move back new op lower bound from 7315524233382809600 to 7315524233265143808. Current Snapshot: MvccSnapshot[applied={T|T < 7315524233382809600}]
I20260806 11:44:18.557363 21362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:18.557538 21362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:18.559856 21362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Sending DeleteTablet for 3 replicas of tablet 7dcd96109d9e4e32a65ca8523bdd9d5c
I20260806 11:44:18.560566 21548 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:18 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:37754
I20260806 11:44:18.560616 21940 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:18 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:46026
I20260806 11:44:18.560927 22091 tablet_replica.cc:330] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: stopping tablet replica
I20260806 11:44:18.561113 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 21422
I20260806 11:44:18.561105 22091 raft_consensus.cc:2229] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:18.561115 21753 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:18 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:57436
I20260806 11:44:18.561362 22091 raft_consensus.cc:2258] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:18.561540 22092 tablet_replica.cc:330] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: stopping tablet replica
I20260806 11:44:18.563568 22092 raft_consensus.cc:2229] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:44:18.563889 22092 raft_consensus.cc:2258] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:18.568172 22092 ts_tablet_manager.cc:1905] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:18.570264 22093 tablet_replica.cc:330] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: stopping tablet replica
I20260806 11:44:18.570693 22093 raft_consensus.cc:2229] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:18.571185 22092 ts_tablet_manager.cc:1918] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:18.571293 22092 log.cc:1196] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/7dcd96109d9e4e32a65ca8523bdd9d5c
I20260806 11:44:18.571108 22093 raft_consensus.cc:2258] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:18.572147 22092 ts_tablet_manager.cc:1939] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 3af94141ac0f483c83c96d82f7fbf2b9: Deleting consensus metadata
I20260806 11:44:18.572932 22093 ts_tablet_manager.cc:1905] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:18.573359 21342 catalog_manager.cc:4841] TS 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647): tablet 7dcd96109d9e4e32a65ca8523bdd9d5c (table test-table [id=63f63a2369c040d79bf0a48f7c9311df]) successfully deleted
W20260806 11:44:18.575582 21342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:40557 (error 108)
I20260806 11:44:18.576167 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 21652
I20260806 11:44:18.581264 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.580942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:40557 (local address 127.0.0.1:37770)
0806 11:44:18.581095 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:18.581172 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 11:44:18.581487 21342 catalog_manager.cc:4584] TS 30ecea34079842c8a5db2dc5061f7f98 (127.0.113.1:40557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7dcd96109d9e4e32a65ca8523bdd9d5c: Network error: Client connection negotiation failed: client connection to 127.0.113.1:40557: connect: Connection refused (error 111)
I20260806 11:44:18.586298 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 21847
I20260806 11:44:18.589149 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:18.586418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:34037 (local address 127.0.0.1:57446)
0806 11:44:18.588964 (+ 2546us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:18.589042 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.2:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2426}
I20260806 11:44:18.597566 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 21311
I20260806 11:44:18.607421 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36125
--webserver_interface=127.0.113.62
--webserver_port=39531
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:18.717185 22095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:18.717463 22095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:18.717551 22095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:18.721487 22095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:18.721558 22095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:18.721596 22095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:18.721649 22095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:18.721685 22095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:18.726852 22095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:18.727818 22095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:18.728839 22095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:18.736991 22108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:18.737002 22103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:18.737002 22104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:18.737177 22095 server_base.cc:1034] running on GCE node
I20260806 11:44:18.738158 22095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:18.739161 22095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:18.740367 22095 hybrid_clock.cc:648] HybridClock initialized: now 1786016658740323 us; error 47 us; skew 500 ppm
Aug 06 11:44:18 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:18.742957 22095 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:44:18.743813 22095 webserver.cc:460] Webserver started at http://127.0.113.62:39531/ using document root <none> and password file <none>
I20260806 11:44:18.744397 22095 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:18.744467 22095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:18.747973 22095 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:44:18.750063 22117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:18.751300 22095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 11:44:18.751394 22095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "d9fb2316ddfb4e80acf3684873c1f915"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "d62e7e9276fe31d5c7604a4301ab1e9c"
server_key_iv: "50f18670759dbef1b37c966c3b9964da"
server_key_version: "encryptionkey@0"
I20260806 11:44:18.751866 22095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:18.790745 22095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:18.793583 22095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:18.793740 22095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:18.800838 22095 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:36125
I20260806 11:44:18.801139 22193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:36125 every 8 connection(s)
I20260806 11:44:18.801957 22095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:44:18.804337 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 22095
I20260806 11:44:18.804813 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:40557
--local_ip_for_outbound_sockets=127.0.113.1
--tserver_master_addrs=127.0.113.62:36125
--webserver_port=33971
--webserver_interface=127.0.113.1
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:44:18.806743 22194 sys_catalog.cc:263] Verifying existing consensus state
I20260806 11:44:18.808401 22194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Bootstrap starting.
I20260806 11:44:18.826164 22194 log.cc:824] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:18.837052 22194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:44:18.837419 22194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915: Bootstrap complete.
I20260806 11:44:18.841480 22194 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:18.841846 22194 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fb2316ddfb4e80acf3684873c1f915, State: Initialized, Role: FOLLOWER
I20260806 11:44:18.842350 22194 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:18.842475 22194 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:44:18.842558 22194 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:44:18.842667 22194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 2 FOLLOWER]: Advancing to term 3
I20260806 11:44:18.844558 22194 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:18.844895 22194 leader_election.cc:302] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9fb2316ddfb4e80acf3684873c1f915; no voters:
I20260806 11:44:18.845222 22194 leader_election.cc:288] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 11:44:18.845301 22200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 11:44:18.845494 22200 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [term 3 LEADER]: Becoming Leader. State: Replica: d9fb2316ddfb4e80acf3684873c1f915, State: Running, Role: LEADER
I20260806 11:44:18.845773 22200 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } }
I20260806 11:44:18.846387 22194 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:18.846639 22203 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d9fb2316ddfb4e80acf3684873c1f915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } } }
I20260806 11:44:18.846767 22203 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:18.847586 22206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:44:18.847747 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9fb2316ddfb4e80acf3684873c1f915. Latest consensus state: current_term: 3 leader_uuid: "d9fb2316ddfb4e80acf3684873c1f915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb2316ddfb4e80acf3684873c1f915" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36125 } } }
I20260806 11:44:18.847865 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fb2316ddfb4e80acf3684873c1f915 [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:18.852941 22206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:44:18.853668 22206 catalog_manager.cc:1259] Loaded cluster ID: 1f70bfb759dc401ea7e96ae02461cc87
I20260806 11:44:18.853729 22206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 11:44:18.854720 22206 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1d73bfe9ad google::LogMessage::Fail() at ??:0
@ 0x7f1d73c0188a google::LogMessage::SendToLog() at ??:0
@ 0x7f1d73bfe4de google::LogMessage::Flush() at ??:0
@ 0x7f1d73bffa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1d7e53dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1d7e53ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1d7e53b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1d7e57f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1d75e503d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1d75e50c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1d75e523f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1d75e41512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1d774856db start_thread at ??:0
@ 0x7f1d726fc61f clone at ??:0
W20260806 11:44:18.928162 22196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:18.928439 22196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:18.928486 22196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:18.932842 22196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:18.932884 22196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:18.932941 22196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:44:18.938047 22196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:40557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=33971
--tserver_master_addrs=127.0.113.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:18.939028 22196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:18.940002 22196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:18.949506 22258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:18.949558 22253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:18.949517 22254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:18.949971 22196 server_base.cc:1034] running on GCE node
I20260806 11:44:18.950440 22196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:18.951202 22196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:18.952407 22196 hybrid_clock.cc:648] HybridClock initialized: now 1786016658952382 us; error 31 us; skew 500 ppm
Aug 06 11:44:18 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:18.955760 22196 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:44:18.956871 22196 webserver.cc:460] Webserver started at http://127.0.113.1:33971/ using document root <none> and password file <none>
I20260806 11:44:18.957573 22196 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:18.957674 22196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:18.962580 22196 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:44:18.965258 22269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:18.966773 22196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:44:18.966893 22196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "30ecea34079842c8a5db2dc5061f7f98"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a9e68cfdd52707d40c729aa43b260d2"
server_key_iv: "7a2ba2fd30710d0e68ad1910b5b56c08"
server_key_version: "encryptionkey@0"
I20260806 11:44:18.967600 22196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:18.990041 22196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:18.993844 22196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:18.994066 22196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:18.994789 22196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:18.996177 22280 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 11:44:19.000429 22196 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 11:44:19.000524 22196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 11:44:19.000595 22196 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 11:44:19.002997 22196 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 11:44:19.003068 22196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 11:44:19.003391 22280 tablet_bootstrap.cc:492] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Bootstrap starting.
I20260806 11:44:19.014889 22196 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:40557
I20260806 11:44:19.015108 22424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:40557 every 8 connection(s)
I20260806 11:44:19.016126 22196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:44:19.022850 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 22196
I20260806 11:44:19.023389 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:34037
--local_ip_for_outbound_sockets=127.0.113.2
--tserver_master_addrs=127.0.113.62:36125
--webserver_port=43883
--webserver_interface=127.0.113.2
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:44:19.023607 22280 log.cc:824] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:19.031558 22280 tablet_bootstrap.cc:492] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:44:19.031993 22280 tablet_bootstrap.cc:492] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Bootstrap complete.
I20260806 11:44:19.033692 22280 ts_tablet_manager.cc:1397] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.005s
I20260806 11:44:19.037770 22280 raft_consensus.cc:348] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:19.038450 22280 raft_consensus.cc:729] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ecea34079842c8a5db2dc5061f7f98, State: Initialized, Role: FOLLOWER
I20260806 11:44:19.039037 22280 consensus_queue.cc:260] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:19.040108 22280 ts_tablet_manager.cc:1428] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260806 11:44:19.086530 22431 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:44:19.018523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:48937)
0806 11:44:19.019121 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:19.019179 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:19.020445 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:19.085716 (+ 65271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":163,"threads_started":1}
W20260806 11:44:19.087194 22426 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:36125 (0 consecutive failures): Network error: Failed to ping master at 127.0.113.62:36125: Client connection negotiation failed: client connection to 127.0.113.62:36125: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 11:44:19.087998 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:19.087675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:60121)
0806 11:44:19.087857 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:19.087938 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 11:44:19.088749 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:19.088527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:56889)
0806 11:44:19.088637 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:19.088686 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 11:44:19.089062 22426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 11:44:19.145941 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:19.146196 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:19.146236 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:19.150192 22433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:19.150242 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:19.150323 22433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:44:19.155539 22433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:34037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=43883
--tserver_master_addrs=127.0.113.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:19.156522 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:19.157529 22433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:19.167197 22440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:19.167253 22433 server_base.cc:1034] running on GCE node
W20260806 11:44:19.167204 22443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:19.167218 22448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:19.167939 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:19.168597 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:19.169763 22433 hybrid_clock.cc:648] HybridClock initialized: now 1786016659169748 us; error 29 us; skew 500 ppm
Aug 06 11:44:19 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:19.172798 22433 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:44:19.173802 22433 webserver.cc:460] Webserver started at http://127.0.113.2:43883/ using document root <none> and password file <none>
I20260806 11:44:19.174341 22433 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:19.174404 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:19.178215 22433 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:44:19.180490 22458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:19.181742 22433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:44:19.181895 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "ce65e14303804dc2a1d63f61358d7c6e"
format_stamp: "Formatted at 2026-08-06 11:44:16 on dist-test-slave-45dx"
server_key: "7a0828fec71cef2b80688bf7f34d1fa8"
server_key_iv: "71e5bac2b2488241d3a2d3dba2426b53"
server_key_version: "encryptionkey@0"
I20260806 11:44:19.182451 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:19.196996 22433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:19.200160 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:19.200398 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:19.200995 22433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:19.202082 22468 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260806 11:44:19.206144 22468 ts_tablet_manager.cc:1852] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 11:44:19.206537 22468 ts_tablet_manager.cc:1905] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:19.210273 22468 ts_tablet_manager.cc:1918] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:19.210408 22468 log.cc:1196] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/7dcd96109d9e4e32a65ca8523bdd9d5c
I20260806 11:44:19.210758 22468 ts_tablet_manager.cc:1939] T 7dcd96109d9e4e32a65ca8523bdd9d5c P ce65e14303804dc2a1d63f61358d7c6e: Deleting consensus metadata
I20260806 11:44:19.211082 22433 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 11:44:19.211140 22433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260806 11:44:19.211213 22433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:19.211247 22433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:19.220582 22433 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:34037
I20260806 11:44:19.220623 22612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:34037 every 8 connection(s)
I20260806 11:44:19.221423 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:44:19.223613 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 22433
I20260806 11:44:19.224112 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:44647
--local_ip_for_outbound_sockets=127.0.113.3
--tserver_master_addrs=127.0.113.62:36125
--webserver_port=39267
--webserver_interface=127.0.113.3
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:44:19.224939 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:19.223542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:37483)
0806 11:44:19.224051 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:19.224153 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":165,"threads_started":1}
W20260806 11:44:19.225482 22614 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:36125 (0 consecutive failures): Network error: Failed to ping master at 127.0.113.62:36125: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
I20260806 11:44:19.226095 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:19.225874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:40463)
0806 11:44:19.225988 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:19.226038 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 11:44:19.226747 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:19.226546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:58931)
0806 11:44:19.226649 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:19.226701 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 11:44:19.226926 22614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 11:44:19.333928 22617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:19.334170 22617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:19.334210 22617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:19.337927 22617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:19.337975 22617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:19.338027 22617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:44:19.342933 22617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:44647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=39267
--tserver_master_addrs=127.0.113.62:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:19.343829 22617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:19.344769 22617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:19.353878 22626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:19.353955 22625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:19.353873 22628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:19.354192 22617 server_base.cc:1034] running on GCE node
I20260806 11:44:19.354596 22617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:19.355196 22617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:19.356391 22617 hybrid_clock.cc:648] HybridClock initialized: now 1786016659356353 us; error 46 us; skew 500 ppm
Aug 06 11:44:19 dist-test-slave-45dx krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:19.358999 22617 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:44:19.359831 22617 webserver.cc:460] Webserver started at http://127.0.113.3:39267/ using document root <none> and password file <none>
I20260806 11:44:19.360446 22617 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:19.360541 22617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:19.364125 22617 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:44:19.366112 22638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:19.367239 22617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:44:19.367355 22617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "3af94141ac0f483c83c96d82f7fbf2b9"
format_stamp: "Formatted at 2026-08-06 11:44:17 on dist-test-slave-45dx"
server_key: "0e0a7040e20f99615f54fb3e9a7e66a6"
server_key_iv: "d8e2e55356a7d6bf12a2d61469785d92"
server_key_version: "encryptionkey@0"
I20260806 11:44:19.367846 22617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:19.398963 22617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:19.401995 22617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:19.402201 22617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:19.402808 22617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:19.403630 22617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:19.403676 22617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:19.403741 22617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:19.403774 22617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:19.412612 22617 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:44647
I20260806 11:44:19.412690 22791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:44647 every 8 connection(s)
I20260806 11:44:19.413578 22617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:44:19.417325 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:19.415544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:60047)
0806 11:44:19.415971 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:19.416072 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
W20260806 11:44:19.417784 22792 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:36125 (0 consecutive failures): Network error: Failed to ping master at 127.0.113.62:36125: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
I20260806 11:44:19.418399 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:19.418157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:49325)
0806 11:44:19.418253 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:19.418330 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 11:44:19.418949 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:19.418780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:56215)
0806 11:44:19.418866 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:19.418905 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 11:44:19.419117 22792 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 11:44:19.421741 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 22617
I20260806 11:44:20.090332 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:20.089836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:43391)
0806 11:44:20.090164 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:20.090218 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":128,"threads_started":1}
I20260806 11:44:20.227977 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:20.227564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:48375)
0806 11:44:20.227834 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:20.227887 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260806 11:44:20.419979 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:20.419587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:50083)
0806 11:44:20.419826 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:20.419879 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260806 11:44:20.560986 22801 raft_consensus.cc:482] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:20.561223 22801 raft_consensus.cc:504] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:20.562435 22801 leader_election.cc:288] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647), ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037)
Aug 06 11:44:20 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.2@KRBTEST.COM
Aug 06 11:44:20 dist-test-slave-45dx krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.3@KRBTEST.COM
I20260806 11:44:20.576184 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:20.562879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:43503 (local address 127.0.113.3:44647)
0806 11:44:20.563196 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:20.563226 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:20.563282 (+ 56us) server_negotiation.cc:407] Connection header received
0806 11:44:20.563958 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:20.563964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:20.564296 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:20.564690 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:20.566029 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:20.566942 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:20.568092 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:20.568628 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:20.568800 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:20.571883 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:20.571894 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:20.571904 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:20.571933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:20.573738 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:20.574249 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:20.574254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:20.574260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:20.574380 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:20.574737 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:20.574742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:20.574744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:20.575116 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:20.575250 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:20.575760 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 11:44:20.576010 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
I20260806 11:44:20.576181 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:20.562647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:44647 (local address 127.0.113.1:43503)
0806 11:44:20.562805 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:20.562820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:20.562926 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:20.564727 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:20.564733 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:20.565326 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:20.565851 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:20.565865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:20.567092 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:20.567103 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:20.567968 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:20.567978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:20.568748 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:20.568752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:20.568990 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:20.569727 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:20.569739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:20.571697 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:20.573846 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:20.573851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:20.573854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:20.574136 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:20.574491 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:20.574494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:20.574496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:20.574642 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:20.575238 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:20.575244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:20.575468 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 11:44:20.575643 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 11:44:20.575984 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":206,"spinlock_wait_cycles":2688}
I20260806 11:44:20.576582 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:20.562672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:34037 (local address 127.0.113.1:53877)
0806 11:44:20.563042 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:20.563056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:20.563172 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:20.564728 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:20.564733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:20.565624 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:20.565850 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:20.565864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:20.567086 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:20.567091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:20.567964 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:20.567973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:20.568741 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:20.568745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:20.568992 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:20.569661 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:20.569678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:20.571258 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:20.573744 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:20.573750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:20.573763 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:20.574103 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:20.574467 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:20.574470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:20.574472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:20.574635 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:20.575238 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:20.575244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:20.575471 (+ 227us) client_negotiation.cc:769] Sending connection context
0806 11:44:20.575641 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 11:44:20.576405 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":546,"spinlock_wait_cycles":3840,"thread_start_us":66,"threads_started":1}
I20260806 11:44:20.576737 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:20.562914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:53877 (local address 127.0.113.2:34037)
0806 11:44:20.563347 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:20.563357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:20.563388 (+ 31us) server_negotiation.cc:407] Connection header received
0806 11:44:20.564036 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:20.564041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:20.564348 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:20.564679 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:20.566022 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:20.566939 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:20.568093 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:20.568617 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:20.568794 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:20.571476 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:20.571484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:20.571501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:20.571535 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:20.573623 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:20.574218 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:20.574224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:20.574229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:20.574362 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:20.574729 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:20.574733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:20.574735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:20.575116 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:20.575250 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:20.576022 (+ 772us) server_negotiation.cc:299] Negotiation successful
0806 11:44:20.576557 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":63,"threads_started":1}
W20260806 11:44:20.579126 22270 leader_election.cc:341] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037): Not found: Tablet not found: 7dcd96109d9e4e32a65ca8523bdd9d5c
W20260806 11:44:20.579281 22274 leader_election.cc:341] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647): Not found: Tablet not found: 7dcd96109d9e4e32a65ca8523bdd9d5c
I20260806 11:44:20.579365 22270 leader_election.cc:302] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ecea34079842c8a5db2dc5061f7f98; no voters: 3af94141ac0f483c83c96d82f7fbf2b9, ce65e14303804dc2a1d63f61358d7c6e
I20260806 11:44:20.579644 22801 raft_consensus.cc:2726] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:44:21.091413 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:21.091006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:57429)
0806 11:44:21.091258 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:21.091314 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260806 11:44:21.229293 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:21.228787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:34721)
0806 11:44:21.229117 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:21.229178 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":107,"threads_started":1}
I20260806 11:44:21.421105 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:21.420697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:33169)
0806 11:44:21.420950 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:21.421019 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260806 11:44:22.092808 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:22.092201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:36979)
0806 11:44:22.092589 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:22.092668 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":135,"threads_started":1}
I20260806 11:44:22.230418 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:22.230008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:49487)
0806 11:44:22.230246 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:22.230308 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260806 11:44:22.422326 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:22.421863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:57741)
0806 11:44:22.422158 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:22.422215 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":102,"threads_started":1}
I20260806 11:44:22.661921 22819 raft_consensus.cc:482] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:22.662058 22819 raft_consensus.cc:504] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:22.662560 22819 leader_election.cc:288] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647), ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037)
W20260806 11:44:22.663414 22270 leader_election.cc:341] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037): Not found: Tablet not found: 7dcd96109d9e4e32a65ca8523bdd9d5c
W20260806 11:44:22.663559 22274 leader_election.cc:341] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647): Not found: Tablet not found: 7dcd96109d9e4e32a65ca8523bdd9d5c
I20260806 11:44:22.663652 22274 leader_election.cc:302] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ecea34079842c8a5db2dc5061f7f98; no voters: 3af94141ac0f483c83c96d82f7fbf2b9, ce65e14303804dc2a1d63f61358d7c6e
I20260806 11:44:22.663833 22819 raft_consensus.cc:2726] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:44:23.094050 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:23.093555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:43281)
0806 11:44:23.093894 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:23.093951 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":79,"threads_started":1}
I20260806 11:44:23.231587 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:23.231187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:56941)
0806 11:44:23.231441 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:23.231500 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260806 11:44:23.423462 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:23.423077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:54051)
0806 11:44:23.423313 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:23.423370 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260806 11:44:24.095525 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:24.095016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:55565)
0806 11:44:24.095365 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:24.095423 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":106,"threads_started":1}
I20260806 11:44:24.232779 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:24.232374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:56721)
0806 11:44:24.232620 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:24.232672 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260806 11:44:24.424706 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:24.424251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:42227)
0806 11:44:24.424564 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:24.424615 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":140,"threads_started":1}
I20260806 11:44:24.562749 22830 raft_consensus.cc:482] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:24.562892 22830 raft_consensus.cc:504] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:24.563474 22830 leader_election.cc:288] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647), ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037)
W20260806 11:44:24.564352 22274 leader_election.cc:341] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647): Not found: Tablet not found: 7dcd96109d9e4e32a65ca8523bdd9d5c
W20260806 11:44:24.564450 22270 leader_election.cc:341] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037): Not found: Tablet not found: 7dcd96109d9e4e32a65ca8523bdd9d5c
I20260806 11:44:24.564517 22270 leader_election.cc:302] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ecea34079842c8a5db2dc5061f7f98; no voters: 3af94141ac0f483c83c96d82f7fbf2b9, ce65e14303804dc2a1d63f61358d7c6e
I20260806 11:44:24.564656 22830 raft_consensus.cc:2726] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:44:25.096844 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:25.096417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:54807)
0806 11:44:25.096688 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:25.096746 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":70,"threads_started":1}
I20260806 11:44:25.233978 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:25.233556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:44373)
0806 11:44:25.233815 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:25.233876 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260806 11:44:25.425983 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:25.425505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:44025)
0806 11:44:25.425816 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:25.425881 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260806 11:44:26.098155 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:26.097661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:34025)
0806 11:44:26.097985 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:26.098050 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":81,"threads_started":1}
I20260806 11:44:26.235294 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:26.234729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:38149)
0806 11:44:26.235064 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:26.235141 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":82,"threads_started":1}
I20260806 11:44:26.427354 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:26.426896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:56863)
0806 11:44:26.427203 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:26.427258 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":77,"threads_started":1}
I20260806 11:44:27.099524 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:27.099051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:49635)
0806 11:44:27.099352 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:27.099417 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
I20260806 11:44:27.236807 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:27.236283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:39913)
0806 11:44:27.236648 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:27.236709 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
I20260806 11:44:27.428632 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:27.428044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:42225)
0806 11:44:27.428477 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:27.428536 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":94,"threads_started":1}
I20260806 11:44:28.100956 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:28.100522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:60979)
0806 11:44:28.100796 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:28.100855 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
I20260806 11:44:28.238109 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:28.237656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:51777)
0806 11:44:28.237947 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:28.238007 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":98,"threads_started":1}
I20260806 11:44:28.429847 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:28.429377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:49117)
0806 11:44:28.429651 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:28.429704 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
I20260806 11:44:28.705962 22853 raft_consensus.cc:482] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:28.706130 22853 raft_consensus.cc:504] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af94141ac0f483c83c96d82f7fbf2b9" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 44647 } } peers { permanent_uuid: "ce65e14303804dc2a1d63f61358d7c6e" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 34037 } } peers { permanent_uuid: "30ecea34079842c8a5db2dc5061f7f98" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 40557 } }
I20260806 11:44:28.706729 22853 leader_election.cc:288] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647), ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037)
W20260806 11:44:28.707783 22274 leader_election.cc:341] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3af94141ac0f483c83c96d82f7fbf2b9 (127.0.113.3:44647): Not found: Tablet not found: 7dcd96109d9e4e32a65ca8523bdd9d5c
W20260806 11:44:28.707933 22270 leader_election.cc:341] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce65e14303804dc2a1d63f61358d7c6e (127.0.113.2:34037): Not found: Tablet not found: 7dcd96109d9e4e32a65ca8523bdd9d5c
I20260806 11:44:28.708037 22270 leader_election.cc:302] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ecea34079842c8a5db2dc5061f7f98; no voters: 3af94141ac0f483c83c96d82f7fbf2b9, ce65e14303804dc2a1d63f61358d7c6e
I20260806 11:44:28.708189 22853 raft_consensus.cc:2726] T 7dcd96109d9e4e32a65ca8523bdd9d5c P 30ecea34079842c8a5db2dc5061f7f98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:44:29.102630 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:29.101888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:52551)
0806 11:44:29.102421 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:29.102477 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":356,"threads_started":1}
I20260806 11:44:29.239549 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:29.238973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:40925)
0806 11:44:29.239370 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:29.239440 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":88,"threads_started":1}
I20260806 11:44:29.431057 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:29.430654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:41215)
0806 11:44:29.430906 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:29.430961 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260806 11:44:30.103870 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:30.103453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:33519)
0806 11:44:30.103719 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:30.103771 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260806 11:44:30.240773 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:30.240394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:40677)
0806 11:44:30.240613 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:30.240665 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":70,"threads_started":1}
I20260806 11:44:30.432224 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:30.431770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:49705)
0806 11:44:30.432071 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:30.432128 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":105,"threads_started":1}
I20260806 11:44:31.105135 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:31.104689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:60265)
0806 11:44:31.104987 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:31.105038 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":123,"threads_started":1}
I20260806 11:44:31.241840 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:31.241446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:43043)
0806 11:44:31.241688 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:31.241738 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
I20260806 11:44:31.433538 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:31.433037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:40873)
0806 11:44:31.433361 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:31.433431 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":90,"threads_started":1}
I20260806 11:44:32.106376 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:32.105893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:48209)
0806 11:44:32.106198 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:32.106258 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":107,"threads_started":1}
I20260806 11:44:32.242914 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:32.242515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:52621)
0806 11:44:32.242769 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:32.242823 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260806 11:44:32.434726 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:32.434332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:41885)
0806 11:44:32.434574 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:32.434628 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260806 11:44:33.107560 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:33.107171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:57849)
0806 11:44:33.107413 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:33.107463 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260806 11:44:33.244120 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:33.243675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:45951)
0806 11:44:33.243962 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:33.244021 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":106,"threads_started":1}
I20260806 11:44:33.435927 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:33.435496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:49463)
0806 11:44:33.435766 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:33.435821 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
I20260806 11:44:34.108932 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:34.108494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.1:34433)
0806 11:44:34.108771 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:34.108833 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260806 11:44:34.245337 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:34.244977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.2:57541)
0806 11:44:34.245200 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:34.245249 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260806 11:44:34.428617 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 22196
I20260806 11:44:34.437121 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:34.436737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36125 (local address 127.0.113.3:40301)
0806 11:44:34.436975 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:34.437032 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260806 11:44:34.441967 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 22433
I20260806 11:44:34.461102 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 22617
2026-08-06T11:44:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19524 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:44:34 dist-test-slave-45dx krb5kdc[22881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:44:34 dist-test-slave-45dx krb5kdc[22881](info): set up 2 sockets
Aug 06 11:44:34 dist-test-slave-45dx krb5kdc[22881](info): commencing operation
krb5kdc: starting...
W20260806 11:44:35.673239 452 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:44:35 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:44:35.801515 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36885
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:35.909253 22902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:35.909499 22902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:35.909538 22902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:35.913461 22902 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 11:44:35.913506 22902 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 11:44:35.913522 22902 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 11:44:35.913537 22902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:35.913594 22902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:35.913630 22902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:35.913664 22902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:35.913698 22902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:35.918749 22902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:35.919749 22902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:35.920765 22902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:35.929292 22910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:35.929311 22912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:35.929311 22909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:35.929502 22902 server_base.cc:1034] running on GCE node
I20260806 11:44:35.930231 22902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:35.931197 22902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:35.932374 22902 hybrid_clock.cc:648] HybridClock initialized: now 1786016675932352 us; error 31 us; skew 500 ppm
Aug 06 11:44:35 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:35.935089 22902 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:44:35.935992 22902 webserver.cc:460] Webserver started at http://127.0.113.62:34917/ using document root <none> and password file <none>
I20260806 11:44:35.936591 22902 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:35.936677 22902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:35.936906 22902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:35.938840 22902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "f367b8bcac544826a0be18e1bb47c40f"
format_stamp: "Formatted at 2026-08-06 11:44:35 on dist-test-slave-45dx"
server_key: "c32af28ed485c4abc26442bff9bad70b"
server_key_iv: "b9d3cd6b805e5a18d1e69a4d589ff025"
server_key_version: "encryptionkey@0"
I20260806 11:44:35.939442 22902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "f367b8bcac544826a0be18e1bb47c40f"
format_stamp: "Formatted at 2026-08-06 11:44:35 on dist-test-slave-45dx"
server_key: "c32af28ed485c4abc26442bff9bad70b"
server_key_iv: "b9d3cd6b805e5a18d1e69a4d589ff025"
server_key_version: "encryptionkey@0"
I20260806 11:44:35.943037 22902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 11:44:35.945369 22922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:35.946496 22902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:44:35.946593 22902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "f367b8bcac544826a0be18e1bb47c40f"
format_stamp: "Formatted at 2026-08-06 11:44:35 on dist-test-slave-45dx"
server_key: "c32af28ed485c4abc26442bff9bad70b"
server_key_iv: "b9d3cd6b805e5a18d1e69a4d589ff025"
server_key_version: "encryptionkey@0"
I20260806 11:44:35.946727 22902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:35.961078 22902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:35.963845 22902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:35.964046 22902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:35.971299 22902 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:36885
I20260806 11:44:35.971611 22990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:36885 every 8 connection(s)
I20260806 11:44:35.972265 22902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:44:35.974918 22992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:35.977463 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 22902
I20260806 11:44:35.977561 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:44:35.977841 452 external_mini_cluster.cc:1428] Setting key e900d8a4feafee81e84e6895d390fd21
I20260806 11:44:35.980656 22992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Bootstrap starting.
I20260806 11:44:35.982749 22992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:35.983494 22992 log.cc:824] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:35.985473 22992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: No bootstrap required, opened a new log
Aug 06 11:44:35 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:35.988970 22992 raft_consensus.cc:348] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:44:35.989171 22992 raft_consensus.cc:374] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:35.989229 22992 raft_consensus.cc:729] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f367b8bcac544826a0be18e1bb47c40f, State: Initialized, Role: FOLLOWER
I20260806 11:44:35.989646 22992 consensus_queue.cc:260] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:44:35.989801 22992 raft_consensus.cc:388] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:44:35.989930 22992 raft_consensus.cc:482] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:44:35.990041 22992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:35.991101 22992 raft_consensus.cc:504] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:44:35.991488 22992 leader_election.cc:302] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f367b8bcac544826a0be18e1bb47c40f; no voters:
I20260806 11:44:35.991858 22992 leader_election.cc:288] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:44:35.991968 23002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:35.992238 23002 raft_consensus.cc:686] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 1 LEADER]: Becoming Leader. State: Replica: f367b8bcac544826a0be18e1bb47c40f, State: Running, Role: LEADER
I20260806 11:44:35.992623 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:35.979236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.0.113.62:36885)
0806 11:44:35.979659 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:35.979673 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:35.979706 (+ 33us) server_negotiation.cc:407] Connection header received
0806 11:44:35.980299 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:35.980377 (+ 78us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:35.980683 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:35.981017 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:35.981819 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:35.982772 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:35.983610 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:35.984225 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:35.984447 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:35.987115 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:35.987120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:35.987132 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:35.987158 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:35.989488 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:35.989924 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:35.989930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:35.989938 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:35.990084 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:35.990412 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:35.990415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:35.990417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:35.990906 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:35.991106 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:35.991293 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 11:44:35.991602 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":128,"threads_started":1}
I20260806 11:44:35.992612 23002 consensus_queue.cc:237] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:44:35.993116 22992 sys_catalog.cc:564] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:35.994473 23003 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f367b8bcac544826a0be18e1bb47c40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } } }
I20260806 11:44:35.994601 23003 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:35.994985 23004 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f367b8bcac544826a0be18e1bb47c40f. Latest consensus state: current_term: 1 leader_uuid: "f367b8bcac544826a0be18e1bb47c40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } } }
I20260806 11:44:35.995124 23004 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: This master's current role is: LEADER
W20260806 11:44:35.998183 23020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:44:35.998257 23020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:44:35.998345 23015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:44:35.999051 23015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:44:36.004261 23015 catalog_manager.cc:1347] Generated new cluster ID: c22d92cc466f4e019c66acfe64023c05
I20260806 11:44:36.004365 23015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:44:36.025902 23015 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:44:36.026785 23015 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:44:36.031394 23015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Generated new TSK 0
I20260806 11:44:36.032049 23015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:44:36.069383 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36885
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:36.179638 23029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:36.179896 23029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:36.179937 23029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:36.183930 23029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:36.183991 23029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:36.184093 23029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:44:36.189278 23029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:36.190307 23029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:36.191334 23029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:36.201169 23045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:36.201180 23040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:36.201364 23029 server_base.cc:1034] running on GCE node
W20260806 11:44:36.201180 23039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:36.201948 23029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:36.202507 23029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:36.203699 23029 hybrid_clock.cc:648] HybridClock initialized: now 1786016676203686 us; error 25 us; skew 500 ppm
Aug 06 11:44:36 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:36.206575 23029 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:44:36.207381 23029 webserver.cc:460] Webserver started at http://127.0.113.1:34249/ using document root <none> and password file <none>
I20260806 11:44:36.207947 23029 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:36.208041 23029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:36.208266 23029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:36.210214 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "25321ea43be8487fafa31f3b55e7dc67"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "0f8995d8bc1f26bd4f4b2335ae410ece"
server_key_iv: "8a98c86c4d3f133ceb58391550445f37"
server_key_version: "encryptionkey@0"
I20260806 11:44:36.210767 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "25321ea43be8487fafa31f3b55e7dc67"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "0f8995d8bc1f26bd4f4b2335ae410ece"
server_key_iv: "8a98c86c4d3f133ceb58391550445f37"
server_key_version: "encryptionkey@0"
I20260806 11:44:36.214319 23029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 11:44:36.216775 23052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:36.217842 23029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:44:36.217972 23029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "25321ea43be8487fafa31f3b55e7dc67"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "0f8995d8bc1f26bd4f4b2335ae410ece"
server_key_iv: "8a98c86c4d3f133ceb58391550445f37"
server_key_version: "encryptionkey@0"
I20260806 11:44:36.218076 23029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:36.244793 23029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:36.247789 23029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:36.248000 23029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:36.248672 23029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:36.249563 23029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:36.249642 23029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:36.249713 23029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:36.249738 23029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:36.259009 23029 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:41043
I20260806 11:44:36.259037 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:41043 every 8 connection(s)
I20260806 11:44:36.259883 23029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:44:36.266168 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 23029
I20260806 11:44:36.266330 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:44:36.266723 452 external_mini_cluster.cc:1428] Setting key 25a3bff296350c976561091f846b24e4
Aug 06 11:44:36 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:36.276136 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.261923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:60815 (local address 127.0.113.62:36885)
0806 11:44:36.262076 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:36.262080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:36.263273 (+ 1193us) server_negotiation.cc:407] Connection header received
0806 11:44:36.264657 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:36.264660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:36.264732 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:36.264845 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:36.266508 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.267232 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.268923 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.269322 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.269414 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.272408 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:36.272412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:36.272414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:36.272446 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:36.274124 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:36.274627 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:36.274631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:36.274633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:36.274723 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:36.275067 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:36.275070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:36.275072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:36.275247 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:36.275355 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:36.275833 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 11:44:36.275965 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 11:44:36.276899 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.262283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:60815)
0806 11:44:36.263086 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:36.263148 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:36.264391 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:36.265051 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:36.265061 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:36.265593 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:36.266301 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.266322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.267376 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.267381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.268773 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.268785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.269622 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.269646 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.269854 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.270578 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:36.270601 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:36.272215 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:36.274240 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:36.274245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:36.274259 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:36.274517 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:36.274823 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:36.274826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:36.274828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:36.274966 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:36.275360 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:36.275368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:36.275600 (+ 232us) client_negotiation.cc:769] Sending connection context
0806 11:44:36.275777 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 11:44:36.276083 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:44:36.278086 23213 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36885
I20260806 11:44:36.278290 23213 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:36.278767 23213 heartbeater.cc:511] Master 127.0.113.62:36885 requested a full tablet report, sending...
I20260806 11:44:36.280597 22944 ts_manager.cc:194] Registered new tserver with Master: 25321ea43be8487fafa31f3b55e7dc67 (127.0.113.1:41043)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:44:36.281801 22944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:60815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:44:36.295536 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36885
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:36.405447 23221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:36.405714 23221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:36.405754 23221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:36.409636 23221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:36.409687 23221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:36.409742 23221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:44:36.414871 23221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:36.415825 23221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:36.416868 23221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:36.426486 23230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:36.426544 23226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:36.426553 23228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:36.426765 23221 server_base.cc:1034] running on GCE node
I20260806 11:44:36.427139 23221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:36.427736 23221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:36.428920 23221 hybrid_clock.cc:648] HybridClock initialized: now 1786016676428903 us; error 32 us; skew 500 ppm
Aug 06 11:44:36 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:36.431710 23221 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:44:36.432653 23221 webserver.cc:460] Webserver started at http://127.0.113.2:44221/ using document root <none> and password file <none>
I20260806 11:44:36.433200 23221 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:36.433298 23221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:36.433600 23221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:36.435608 23221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "5b798d8f839e4933b6af83e607fab598"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "9c1fcfcf2ae9cc5a791d6018a4aaeaab"
server_key_iv: "f46ac3ba75786c0c347c0f789fd6b616"
server_key_version: "encryptionkey@0"
I20260806 11:44:36.436201 23221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "5b798d8f839e4933b6af83e607fab598"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "9c1fcfcf2ae9cc5a791d6018a4aaeaab"
server_key_iv: "f46ac3ba75786c0c347c0f789fd6b616"
server_key_version: "encryptionkey@0"
I20260806 11:44:36.439934 23221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:44:36.442515 23243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:36.443691 23221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:44:36.443831 23221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "5b798d8f839e4933b6af83e607fab598"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "9c1fcfcf2ae9cc5a791d6018a4aaeaab"
server_key_iv: "f46ac3ba75786c0c347c0f789fd6b616"
server_key_version: "encryptionkey@0"
I20260806 11:44:36.443944 23221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:36.455336 23221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:36.458088 23221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:36.458300 23221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:36.458843 23221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:36.459619 23221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:36.459693 23221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:36.459759 23221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:36.459794 23221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:36.469282 23221 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:44159
I20260806 11:44:36.469316 23402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:44159 every 8 connection(s)
I20260806 11:44:36.470103 23221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:44:36.471917 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 23221
I20260806 11:44:36.472059 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:44:36.472432 452 external_mini_cluster.cc:1428] Setting key b635e5e500c3e67053374a328e80c081
Aug 06 11:44:36 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:44:36.484850 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.472170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:59543 (local address 127.0.113.62:36885)
0806 11:44:36.472351 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:36.472355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:36.473181 (+ 826us) server_negotiation.cc:407] Connection header received
0806 11:44:36.474180 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:36.474183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:36.474234 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:36.474329 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:36.475383 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.476221 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.478010 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.478254 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.478375 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.480836 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:36.480844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:36.480849 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:36.480884 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:36.482649 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:36.483178 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:36.483185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:36.483187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:36.483286 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:36.483627 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:36.483633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:36.483636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:36.483859 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:36.483969 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:36.484620 (+ 651us) server_negotiation.cc:299] Negotiation successful
0806 11:44:36.484725 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 11:44:36.485639 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.472253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:59543)
0806 11:44:36.473008 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:36.473071 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:36.473973 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:36.474454 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:36.474474 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:36.474799 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:36.475232 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.475245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.476394 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.476401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.477871 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.477882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.478410 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.478415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.478611 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.479224 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:36.479239 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:36.480657 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:36.482776 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:36.482781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:36.482791 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:36.483060 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:36.483393 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:36.483395 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:36.483397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:36.483525 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:36.484019 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:36.484030 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:36.484341 (+ 311us) client_negotiation.cc:769] Sending connection context
0806 11:44:36.484548 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 11:44:36.484790 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":159,"threads_started":1}
I20260806 11:44:36.486922 23405 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
I20260806 11:44:36.487167 23405 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:36.487636 23405 heartbeater.cc:511] Master 127.0.113.62:36885 requested a full tablet report, sending...
I20260806 11:44:36.488724 22944 ts_manager.cc:194] Registered new tserver with Master: 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159)
I20260806 11:44:36.489282 22944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:59543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:44:36.503219 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36885
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:36.615749 23413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:36.616056 23413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:36.616101 23413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:36.620128 23413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:36.620188 23413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:36.620296 23413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:44:36.625658 23413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:36.626662 23413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:36.627658 23413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:36.637300 23422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:36.637420 23413 server_base.cc:1034] running on GCE node
W20260806 11:44:36.637310 23425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:36.637480 23423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:36.638034 23413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:36.638671 23413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:36.639873 23413 hybrid_clock.cc:648] HybridClock initialized: now 1786016676639818 us; error 76 us; skew 500 ppm
Aug 06 11:44:36 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:36.642719 23413 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:44:36.643630 23413 webserver.cc:460] Webserver started at http://127.0.113.3:39347/ using document root <none> and password file <none>
I20260806 11:44:36.644178 23413 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:36.644272 23413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:36.644534 23413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:44:36.646500 23413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "b55f9dc9fd7a42838e66f008f94be29e"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "f1c2a1d0fd872134eeed2541ef68dcd8"
server_key_iv: "d8407bef3d55af692c4040b3b0878af4"
server_key_version: "encryptionkey@0"
I20260806 11:44:36.647105 23413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "b55f9dc9fd7a42838e66f008f94be29e"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "f1c2a1d0fd872134eeed2541ef68dcd8"
server_key_iv: "d8407bef3d55af692c4040b3b0878af4"
server_key_version: "encryptionkey@0"
I20260806 11:44:36.651046 23413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 11:44:36.653478 23432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:36.654663 23413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:44:36.654838 23413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "b55f9dc9fd7a42838e66f008f94be29e"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "f1c2a1d0fd872134eeed2541ef68dcd8"
server_key_iv: "d8407bef3d55af692c4040b3b0878af4"
server_key_version: "encryptionkey@0"
I20260806 11:44:36.654956 23413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:36.676550 23413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:36.679996 23413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:36.680208 23413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:36.680855 23413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:36.681703 23413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:36.681802 23413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:36.681892 23413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:36.681943 23413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:36.691298 23413 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:45501
I20260806 11:44:36.691340 23590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:45501 every 8 connection(s)
I20260806 11:44:36.692166 23413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:44:36.700014 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 23413
I20260806 11:44:36.700155 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:44:36.700520 452 external_mini_cluster.cc:1428] Setting key dbe88bfad7ad0b1ec4c70f6bc542f6f2
Aug 06 11:44:36 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:36.705354 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.694067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:34869 (local address 127.0.113.62:36885)
0806 11:44:36.694218 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:36.694222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:36.695362 (+ 1140us) server_negotiation.cc:407] Connection header received
0806 11:44:36.696168 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:36.696171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:36.696234 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:36.696371 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:36.697449 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.697959 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.699026 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.699227 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.699320 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.702152 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:36.702155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:36.702157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:36.702180 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:36.703591 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:36.704050 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:36.704053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:36.704054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:36.704124 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:36.704487 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:36.704489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:36.704491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:36.704657 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:36.704733 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:36.705154 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 11:44:36.705242 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 11:44:36.706076 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.694476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:34869)
0806 11:44:36.695216 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:36.695257 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:36.695997 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:36.696513 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:36.696520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:36.696864 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:36.697315 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.697328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.698079 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.698082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.698920 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.698929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.699366 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.699369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.699500 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.700039 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:36.700063 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:36.701990 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:36.703697 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:36.703702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:36.703713 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:36.703966 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:36.704202 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:36.704204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:36.704206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:36.704394 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:36.704747 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:36.704752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:36.704967 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 11:44:36.705109 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 11:44:36.705322 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":122,"threads_started":1}
I20260806 11:44:36.707103 23591 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36885
I20260806 11:44:36.707374 23591 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:36.707867 23591 heartbeater.cc:511] Master 127.0.113.62:36885 requested a full tablet report, sending...
I20260806 11:44:36.708876 22944 ts_manager.cc:194] Registered new tserver with Master: b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501)
I20260806 11:44:36.709452 22944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:34869
I20260806 11:44:36.714460 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:44:36.723495 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.716481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.0.113.62:36885)
0806 11:44:36.716608 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:36.716612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:36.716827 (+ 215us) server_negotiation.cc:407] Connection header received
0806 11:44:36.716916 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:36.716918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:36.716963 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:36.717030 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:36.717720 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.718196 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.719007 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.719201 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.719272 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.720592 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:36.720594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:36.720596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:36.720618 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:36.722006 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:36.722504 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:36.722506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:36.722507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:36.722579 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:36.722815 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:36.722818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:36.722819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:36.723021 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:36.723121 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:36.723298 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 11:44:36.723380 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 11:44:36.726155 22944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:44:36.729110 22944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:44:36.742916 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.737100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:41043 (local address 127.0.0.1:48002)
0806 11:44:36.737600 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:36.737650 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:36.737777 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:36.738394 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:36.738402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:36.738427 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:36.738565 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.738573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.740033 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.740036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.741407 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.741418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.742373 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.742377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.742486 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.742527 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 11:44:36.742647 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 11:44:36.742766 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260806 11:44:36.742937 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.737359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.0.113.1:41043)
0806 11:44:36.737828 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:36.737844 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:36.737869 (+ 25us) server_negotiation.cc:407] Connection header received
0806 11:44:36.737952 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:36.737961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:36.738196 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:36.738734 (+ 538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:36.738929 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.739902 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.741549 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.742232 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.742356 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.742501 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:36.742689 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 11:44:36.742787 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":105,"threads_started":1}
I20260806 11:44:36.743280 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.737890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:44159 (local address 127.0.0.1:44344)
0806 11:44:36.738158 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:36.738171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:36.738295 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:36.738960 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:36.738965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:36.738984 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:36.739081 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.739086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.740505 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.740509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.741602 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.741613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.742880 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.742883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.742989 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.743020 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 11:44:36.743104 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 11:44:36.743166 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":110,"threads_started":1}
I20260806 11:44:36.743777 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.737606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.0.113.2:44159)
0806 11:44:36.738117 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:36.738128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:36.738395 (+ 267us) server_negotiation.cc:407] Connection header received
0806 11:44:36.738482 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:36.738487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:36.738677 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:36.738822 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:36.739197 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.740363 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.741727 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.742598 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.742693 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.742819 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:36.743543 (+ 724us) server_negotiation.cc:299] Negotiation successful
0806 11:44:36.743640 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":122,"threads_started":1}
I20260806 11:44:36.743734 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.738260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:45501 (local address 127.0.0.1:42118)
0806 11:44:36.739067 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:36.739081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:36.739173 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:36.739667 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:36.739670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:36.739689 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:36.739767 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.739772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.740959 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.740963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.742084 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:36.742094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.743351 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.743356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:36.743467 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.743505 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 11:44:36.743577 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 11:44:36.743639 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":95,"threads_started":1}
I20260806 11:44:36.744112 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:36.738460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.0.113.3:45501)
0806 11:44:36.738885 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:36.738896 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:36.739235 (+ 339us) server_negotiation.cc:407] Connection header received
0806 11:44:36.739326 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:36.739331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:36.739511 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:36.739660 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:36.739886 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.740821 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.742239 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:36.743041 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:36.743160 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:36.743264 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:36.743854 (+ 590us) server_negotiation.cc:299] Negotiation successful
0806 11:44:36.743969 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":145,"threads_started":1}
I20260806 11:44:36.746562 23312 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d (DEFAULT_TABLE table=test-table [id=19de7d4233b943f5b819de319c0da204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:36.746564 23119 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d (DEFAULT_TABLE table=test-table [id=19de7d4233b943f5b819de319c0da204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:36.746814 23499 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d (DEFAULT_TABLE table=test-table [id=19de7d4233b943f5b819de319c0da204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:36.747736 23499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:36.747730 23119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:36.747730 23312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:36.755474 23619 tablet_bootstrap.cc:492] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: Bootstrap starting.
I20260806 11:44:36.756613 23617 tablet_bootstrap.cc:492] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: Bootstrap starting.
I20260806 11:44:36.757853 23619 tablet_bootstrap.cc:654] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:36.758663 23617 tablet_bootstrap.cc:654] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:36.758652 23619 log.cc:824] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:36.759424 23617 log.cc:824] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:36.759994 23618 tablet_bootstrap.cc:492] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Bootstrap starting.
I20260806 11:44:36.760763 23619 tablet_bootstrap.cc:492] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: No bootstrap required, opened a new log
I20260806 11:44:36.760967 23619 ts_tablet_manager.cc:1397] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 11:44:36.761359 23617 tablet_bootstrap.cc:492] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: No bootstrap required, opened a new log
I20260806 11:44:36.761541 23617 ts_tablet_manager.cc:1397] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:44:36.762032 23618 tablet_bootstrap.cc:654] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:36.762972 23618 log.cc:824] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:36.764532 23618 tablet_bootstrap.cc:492] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: No bootstrap required, opened a new log
I20260806 11:44:36.764717 23618 ts_tablet_manager.cc:1397] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:44:36.764961 23619 raft_consensus.cc:348] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:36.765261 23619 raft_consensus.cc:374] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:36.765337 23619 raft_consensus.cc:729] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b798d8f839e4933b6af83e607fab598, State: Initialized, Role: FOLLOWER
I20260806 11:44:36.765305 23617 raft_consensus.cc:348] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:36.765560 23617 raft_consensus.cc:374] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:36.765630 23617 raft_consensus.cc:729] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55f9dc9fd7a42838e66f008f94be29e, State: Initialized, Role: FOLLOWER
I20260806 11:44:36.765856 23619 consensus_queue.cc:260] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:36.766001 23617 consensus_queue.cc:260] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:36.766669 23591 heartbeater.cc:503] Master 127.0.113.62:36885 was elected leader, sending a full tablet report...
I20260806 11:44:36.766835 23617 ts_tablet_manager.cc:1428] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 11:44:36.766892 23619 ts_tablet_manager.cc:1428] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 11:44:36.767177 23405 heartbeater.cc:503] Master 127.0.113.62:36885 was elected leader, sending a full tablet report...
I20260806 11:44:36.768424 23618 raft_consensus.cc:348] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:36.768664 23618 raft_consensus.cc:374] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:36.768719 23618 raft_consensus.cc:729] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25321ea43be8487fafa31f3b55e7dc67, State: Initialized, Role: FOLLOWER
I20260806 11:44:36.769119 23618 consensus_queue.cc:260] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:36.769748 23213 heartbeater.cc:503] Master 127.0.113.62:36885 was elected leader, sending a full tablet report...
I20260806 11:44:36.770072 23618 ts_tablet_manager.cc:1428] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260806 11:44:36.943162 23592 tablet.cc:2369] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:44:36.971207 23407 tablet.cc:2369] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:44:37.011389 23214 tablet.cc:2369] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:44:37.037070 23626 raft_consensus.cc:482] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:37.037411 23626 raft_consensus.cc:504] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:37.038575 23626 leader_election.cc:288] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159), b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501)
I20260806 11:44:37.042512 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:37.038957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:50547 (local address 127.0.113.2:44159)
0806 11:44:37.039111 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:37.039116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:37.039244 (+ 128us) server_negotiation.cc:407] Connection header received
0806 11:44:37.039430 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:37.039434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:37.039492 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:37.039570 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:37.039945 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.040479 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.041540 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.041950 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.042072 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:37.042152 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:37.042319 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 11:44:37.042393 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:44:37.042531 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:37.038822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:44159 (local address 127.0.113.1:50547)
0806 11:44:37.039141 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:37.039156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:37.039298 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:37.039642 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:37.039646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:37.039672 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:37.039819 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:37.039825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.040601 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.040604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:37.041405 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:37.041414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.042075 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.042079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:37.042180 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:37.042209 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 11:44:37.042302 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 11:44:37.042371 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"spinlock_wait_cycles":768,"thread_start_us":110,"threads_started":1}
I20260806 11:44:37.043005 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:37.039254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:45501 (local address 127.0.113.1:50339)
0806 11:44:37.039564 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:37.039576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:37.039730 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:37.040026 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:37.040029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:37.040045 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:37.040165 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:37.040171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.040930 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.040935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:37.041708 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:37.041717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.042399 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.042403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:37.042491 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:37.042512 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 11:44:37.042758 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 11:44:37.042831 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":1,"spinlock_wait_cycles":1792,"thread_start_us":89,"threads_started":1}
I20260806 11:44:37.043020 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:37.039385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:50339 (local address 127.0.113.3:45501)
0806 11:44:37.039520 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:37.039562 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:37.039642 (+ 80us) server_negotiation.cc:407] Connection header received
0806 11:44:37.039875 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:37.039877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:37.039923 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:37.040013 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:37.040279 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.040811 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.041865 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.042266 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.042390 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:37.042449 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:37.042809 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 11:44:37.042893 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 11:44:37.043255 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee6d6b9a2dd47ed9cfd7cc25dc4473d" candidate_uuid: "25321ea43be8487fafa31f3b55e7dc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b798d8f839e4933b6af83e607fab598" is_pre_election: true
I20260806 11:44:37.043596 23340 raft_consensus.cc:2454] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25321ea43be8487fafa31f3b55e7dc67 in term 0.
I20260806 11:44:37.043640 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee6d6b9a2dd47ed9cfd7cc25dc4473d" candidate_uuid: "25321ea43be8487fafa31f3b55e7dc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b55f9dc9fd7a42838e66f008f94be29e" is_pre_election: true
I20260806 11:44:37.043906 23529 raft_consensus.cc:2454] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25321ea43be8487fafa31f3b55e7dc67 in term 0.
I20260806 11:44:37.044104 23054 leader_election.cc:302] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25321ea43be8487fafa31f3b55e7dc67, 5b798d8f839e4933b6af83e607fab598; no voters:
I20260806 11:44:37.044421 23626 raft_consensus.cc:2781] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:44:37.044497 23626 raft_consensus.cc:482] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:44:37.044526 23626 raft_consensus.cc:3037] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:37.045567 23626 raft_consensus.cc:504] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:37.045910 23626 leader_election.cc:288] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 1 election: Requested vote from peers 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159), b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501)
I20260806 11:44:37.046380 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee6d6b9a2dd47ed9cfd7cc25dc4473d" candidate_uuid: "25321ea43be8487fafa31f3b55e7dc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b798d8f839e4933b6af83e607fab598"
I20260806 11:44:37.046500 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee6d6b9a2dd47ed9cfd7cc25dc4473d" candidate_uuid: "25321ea43be8487fafa31f3b55e7dc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b55f9dc9fd7a42838e66f008f94be29e"
I20260806 11:44:37.046530 23340 raft_consensus.cc:3037] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:37.046656 23529 raft_consensus.cc:3037] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:37.047649 23340 raft_consensus.cc:2454] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25321ea43be8487fafa31f3b55e7dc67 in term 1.
I20260806 11:44:37.047866 23529 raft_consensus.cc:2454] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25321ea43be8487fafa31f3b55e7dc67 in term 1.
I20260806 11:44:37.047979 23054 leader_election.cc:302] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25321ea43be8487fafa31f3b55e7dc67, 5b798d8f839e4933b6af83e607fab598; no voters:
I20260806 11:44:37.048264 23626 raft_consensus.cc:2781] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:37.048521 23626 raft_consensus.cc:686] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 1 LEADER]: Becoming Leader. State: Replica: 25321ea43be8487fafa31f3b55e7dc67, State: Running, Role: LEADER
I20260806 11:44:37.048863 23626 consensus_queue.cc:237] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:37.052065 22942 catalog_manager.cc:5495] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25321ea43be8487fafa31f3b55e7dc67 (127.0.113.1). New cstate: current_term: 1 leader_uuid: "25321ea43be8487fafa31f3b55e7dc67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } health_report { overall_health: UNKNOWN } } }
I20260806 11:44:37.104265 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:37.100952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.0.113.1:41043)
0806 11:44:37.101089 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:37.101093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:37.101107 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:44:37.101256 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:37.101259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:37.101314 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:37.101400 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:44:37.101612 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.102125 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.102929 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:37.103205 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:37.103280 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:37.103545 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:44:37.103912 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:44:37.103999 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:37.104068 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 11:44:37.104112 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 11:44:37.110266 23340 raft_consensus.cc:1261] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [term 1 FOLLOWER]: Refusing update from remote peer 25321ea43be8487fafa31f3b55e7dc67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:37.110280 23529 raft_consensus.cc:1261] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [term 1 FOLLOWER]: Refusing update from remote peer 25321ea43be8487fafa31f3b55e7dc67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:37.111006 23629 consensus_queue.cc:1035] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:37.111205 23626 consensus_queue.cc:1035] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:37.119064 23632 mvcc.cc:205] Tried to move back new op lower bound from 7315524309435822080 to 7315524309194842112. Current Snapshot: MvccSnapshot[applied={T|T < 7315524309435822080}]
I20260806 11:44:37.121310 23633 mvcc.cc:205] Tried to move back new op lower bound from 7315524309435822080 to 7315524309194842112. Current Snapshot: MvccSnapshot[applied={T|T < 7315524309435822080}]
I20260806 11:44:37.121709 23635 mvcc.cc:205] Tried to move back new op lower bound from 7315524309435822080 to 7315524309194842112. Current Snapshot: MvccSnapshot[applied={T|T < 7315524309435822080}]
I20260806 11:44:37.129778 22944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:37.129973 22944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:37.132412 22944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Sending DeleteTablet for 3 replicas of tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d
I20260806 11:44:37.133152 23119 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:37 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:48002
I20260806 11:44:37.133332 23312 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:37 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:44344
I20260806 11:44:37.133517 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 23029
I20260806 11:44:37.133641 23499 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:37 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:42118
I20260806 11:44:37.142441 23647 tablet_replica.cc:330] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: stopping tablet replica
I20260806 11:44:37.142822 23647 raft_consensus.cc:2229] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:37.143155 23647 raft_consensus.cc:2258] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:37.144536 23646 tablet_replica.cc:330] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: stopping tablet replica
I20260806 11:44:37.144879 23647 ts_tablet_manager.cc:1905] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:37.144906 23646 raft_consensus.cc:2229] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:37.145360 23646 raft_consensus.cc:2258] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:37.146625 23646 ts_tablet_manager.cc:1905] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:37.148682 23647 ts_tablet_manager.cc:1918] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:37.148793 23647 log.cc:1196] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/8ee6d6b9a2dd47ed9cfd7cc25dc4473d
I20260806 11:44:37.149179 23647 ts_tablet_manager.cc:1939] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P b55f9dc9fd7a42838e66f008f94be29e: Deleting consensus metadata
I20260806 11:44:37.149502 23646 ts_tablet_manager.cc:1918] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:37.149595 23646 log.cc:1196] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/8ee6d6b9a2dd47ed9cfd7cc25dc4473d
I20260806 11:44:37.149879 23646 ts_tablet_manager.cc:1939] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 5b798d8f839e4933b6af83e607fab598: Deleting consensus metadata
I20260806 11:44:37.150523 22923 catalog_manager.cc:4841] TS b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501): tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d (table test-table [id=19de7d4233b943f5b819de319c0da204]) successfully deleted
I20260806 11:44:37.151031 22923 catalog_manager.cc:4841] TS 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159): tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d (table test-table [id=19de7d4233b943f5b819de319c0da204]) successfully deleted
I20260806 11:44:37.152119 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 23221
W20260806 11:44:37.152258 22926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:41043 (error 108)
I20260806 11:44:37.160058 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:37.159713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:41043 (local address 127.0.0.1:48008)
0806 11:44:37.159857 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:37.159952 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:41043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 11:44:37.160246 22926 catalog_manager.cc:4584] TS 25321ea43be8487fafa31f3b55e7dc67 (127.0.113.1:41043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d: Network error: Client connection negotiation failed: client connection to 127.0.113.1:41043: connect: Connection refused (error 111)
I20260806 11:44:37.165150 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 23413
I20260806 11:44:37.175388 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 22902
I20260806 11:44:57.186021 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36885
--webserver_interface=127.0.113.62
--webserver_port=34917
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:44:57.300132 23650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:57.300426 23650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:57.300506 23650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:57.304491 23650 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 11:44:57.304544 23650 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 11:44:57.304561 23650 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 11:44:57.304575 23650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:44:57.304594 23650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:57.304612 23650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:57.304677 23650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:44:57.304703 23650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:44:57.309830 23650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=34917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:57.310931 23650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:57.311967 23650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:57.326066 23661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:57.326086 23663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:57.326056 23667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:57.326113 23650 server_base.cc:1034] running on GCE node
I20260806 11:44:57.327050 23650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:57.328245 23650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:57.329463 23650 hybrid_clock.cc:648] HybridClock initialized: now 1786016697329434 us; error 43 us; skew 500 ppm
Aug 06 11:44:57 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:57.340420 23650 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:44:57.341457 23650 webserver.cc:460] Webserver started at http://127.0.113.62:34917/ using document root <none> and password file <none>
I20260806 11:44:57.342120 23650 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:57.342239 23650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:57.346413 23650 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 11:44:57.348599 23675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:57.349839 23650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:44:57.349988 23650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "f367b8bcac544826a0be18e1bb47c40f"
format_stamp: "Formatted at 2026-08-06 11:44:35 on dist-test-slave-45dx"
server_key: "c32af28ed485c4abc26442bff9bad70b"
server_key_iv: "b9d3cd6b805e5a18d1e69a4d589ff025"
server_key_version: "encryptionkey@0"
I20260806 11:44:57.350629 23650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:57.385183 23650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:57.388252 23650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:57.388475 23650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:57.395916 23650 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:36885
I20260806 11:44:57.396239 23745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:36885 every 8 connection(s)
I20260806 11:44:57.397055 23650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:44:57.401427 23746 sys_catalog.cc:263] Verifying existing consensus state
I20260806 11:44:57.403148 23746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Bootstrap starting.
I20260806 11:44:57.404242 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 23650
I20260806 11:44:57.404757 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:41043
--local_ip_for_outbound_sockets=127.0.113.1
--tserver_master_addrs=127.0.113.62:36885
--webserver_port=34249
--webserver_interface=127.0.113.1
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:44:57.420153 23746 log.cc:824] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:57.428864 23746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:44:57.429239 23746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Bootstrap complete.
I20260806 11:44:57.433188 23746 raft_consensus.cc:348] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:44:57.433616 23746 raft_consensus.cc:729] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f367b8bcac544826a0be18e1bb47c40f, State: Initialized, Role: FOLLOWER
I20260806 11:44:57.434131 23746 consensus_queue.cc:260] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:44:57.434275 23746 raft_consensus.cc:388] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:44:57.434370 23746 raft_consensus.cc:482] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:44:57.434496 23746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 1 FOLLOWER]: Advancing to term 2
I20260806 11:44:57.436260 23746 raft_consensus.cc:504] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:44:57.436647 23746 leader_election.cc:302] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f367b8bcac544826a0be18e1bb47c40f; no voters:
I20260806 11:44:57.436983 23746 leader_election.cc:288] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 11:44:57.437069 23757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 2 FOLLOWER]: Leader election won for term 2
I20260806 11:44:57.437273 23757 raft_consensus.cc:686] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 2 LEADER]: Becoming Leader. State: Replica: f367b8bcac544826a0be18e1bb47c40f, State: Running, Role: LEADER
I20260806 11:44:57.437546 23757 consensus_queue.cc:237] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:44:57.438155 23746 sys_catalog.cc:564] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:44:57.438431 23757 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f367b8bcac544826a0be18e1bb47c40f. Latest consensus state: current_term: 2 leader_uuid: "f367b8bcac544826a0be18e1bb47c40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } } }
I20260806 11:44:57.438651 23757 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:57.438406 23760 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f367b8bcac544826a0be18e1bb47c40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } } }
I20260806 11:44:57.438985 23760 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: This master's current role is: LEADER
I20260806 11:44:57.439786 23763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:44:57.444501 23763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:44:57.445137 23763 catalog_manager.cc:1259] Loaded cluster ID: c22d92cc466f4e019c66acfe64023c05
I20260806 11:44:57.445196 23763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:44:57.447618 23763 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:44:57.448307 23763 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Loaded TSK: 0
I20260806 11:44:57.466024 23763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Generated new TSK 1
I20260806 11:44:57.466256 23763 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 11:44:57.523814 23750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:57.524052 23750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:57.524094 23750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:57.528173 23750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:57.528235 23750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:57.528291 23750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:44:57.533416 23750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:41043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=34249
--tserver_master_addrs=127.0.113.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:57.534433 23750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:57.535431 23750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:57.546178 23791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:57.546173 23795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:57.546352 23792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:57.546576 23750 server_base.cc:1034] running on GCE node
I20260806 11:44:57.547006 23750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:57.547672 23750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:57.548883 23750 hybrid_clock.cc:648] HybridClock initialized: now 1786016697548870 us; error 30 us; skew 500 ppm
Aug 06 11:44:57 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:57.551918 23750 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:44:57.552904 23750 webserver.cc:460] Webserver started at http://127.0.113.1:34249/ using document root <none> and password file <none>
I20260806 11:44:57.553505 23750 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:57.553623 23750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:57.557616 23750 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 11:44:57.559811 23804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:57.561108 23750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:44:57.561203 23750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "25321ea43be8487fafa31f3b55e7dc67"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "0f8995d8bc1f26bd4f4b2335ae410ece"
server_key_iv: "8a98c86c4d3f133ceb58391550445f37"
server_key_version: "encryptionkey@0"
I20260806 11:44:57.561826 23750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:57.583441 23750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:57.586436 23750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:57.586642 23750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:57.587230 23750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:57.588397 23813 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 11:44:57.592284 23750 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 11:44:57.592410 23750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260806 11:44:57.592492 23750 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 11:44:57.594200 23750 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 11:44:57.594265 23750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 11:44:57.594530 23813 tablet_bootstrap.cc:492] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Bootstrap starting.
I20260806 11:44:57.605322 23750 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:41043
I20260806 11:44:57.605408 23967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:41043 every 8 connection(s)
I20260806 11:44:57.606338 23750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:44:57.612329 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 23750
I20260806 11:44:57.612522 23813 log.cc:824] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:57.612886 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:44159
--local_ip_for_outbound_sockets=127.0.113.2
--tserver_master_addrs=127.0.113.62:36885
--webserver_port=44221
--webserver_interface=127.0.113.2
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:44:57 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:57.621197 23813 tablet_bootstrap.cc:492] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:44:57.621804 23813 tablet_bootstrap.cc:492] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Bootstrap complete.
I20260806 11:44:57.623208 23813 ts_tablet_manager.cc:1397] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20260806 11:44:57.626016 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:57.608454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:38819)
0806 11:44:57.609045 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:57.609086 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:57.609897 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:57.611085 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:57.611093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:57.611544 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:57.612021 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:57.612035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.613949 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.613952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:57.615163 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:57.615180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.616337 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.616343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:57.616607 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:57.617350 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:57.617376 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:57.619328 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:57.622298 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:57.622307 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:57.622323 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:57.622707 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:57.623152 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:57.623156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:57.623157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:57.623346 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:57.624144 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:57.624151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:57.624477 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 11:44:57.624690 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 11:44:57.625024 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":171,"threads_started":1}
I20260806 11:44:57.626125 23813 raft_consensus.cc:348] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:57.626595 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:57.608433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:38819 (local address 127.0.113.62:36885)
0806 11:44:57.608854 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:57.608875 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:57.609203 (+ 328us) server_negotiation.cc:407] Connection header received
0806 11:44:57.610379 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:57.610389 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:57.610667 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:57.611045 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:57.612239 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.613770 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.615432 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.616128 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.616360 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:57.619560 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:57.619569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:57.619588 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:57.619631 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:57.622154 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:57.622844 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:57.622850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:57.622857 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:57.623033 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:57.623464 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:57.623469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:57.623471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:57.624006 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:57.624167 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:57.624852 (+ 685us) server_negotiation.cc:299] Negotiation successful
0806 11:44:57.625195 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":156,"threads_started":1}
I20260806 11:44:57.626679 23813 raft_consensus.cc:729] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25321ea43be8487fafa31f3b55e7dc67, State: Initialized, Role: FOLLOWER
I20260806 11:44:57.627166 23813 consensus_queue.cc:260] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } }
I20260806 11:44:57.628821 23968 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36885
I20260806 11:44:57.628988 23813 ts_tablet_manager.cc:1428] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 11:44:57.629055 23968 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:57.629578 23968 heartbeater.cc:511] Master 127.0.113.62:36885 requested a full tablet report, sending...
I20260806 11:44:57.631958 23704 ts_manager.cc:194] Registered new tserver with Master: 25321ea43be8487fafa31f3b55e7dc67 (127.0.113.1:41043)
I20260806 11:44:57.635231 23704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:38819
I20260806 11:44:57.637969 23968 heartbeater.cc:503] Master 127.0.113.62:36885 was elected leader, sending a full tablet report...
Aug 06 11:44:57 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.1@KRBTEST.COM
I20260806 11:44:57.645427 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:57.634679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:41043 (local address 127.0.0.1:35206)
0806 11:44:57.635246 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:57.635291 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:57.635505 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:57.636021 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:57.636027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:57.636476 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:57.636580 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:57.636586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.637540 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.637547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:57.638297 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:57.638313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.639160 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.639164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:57.639266 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:57.640013 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:57.640025 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:57.641680 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:57.643744 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:57.643752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:57.643757 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:57.643982 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:57.644289 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:57.644291 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:57.644292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:57.644435 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:57.644942 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:57.644946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:57.645042 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 11:44:57.645144 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 11:44:57.645288 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":97,"threads_started":1}
I20260806 11:44:57.645601 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:57.634776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.0.113.1:41043)
0806 11:44:57.635410 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:57.635421 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:57.635444 (+ 23us) server_negotiation.cc:407] Connection header received
0806 11:44:57.635661 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:57.635667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:57.635853 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:57.636002 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:57.636702 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.637414 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.638430 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.638969 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.639142 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:57.641851 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:57.641861 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:57.641874 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:57.641900 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:57.643632 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:57.644091 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:57.644097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:57.644099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:57.644196 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:57.644525 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:57.644528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:57.644530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:57.644829 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:57.644962 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:57.645273 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 11:44:57.645446 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":123,"threads_started":1}
I20260806 11:44:57.646489 23874 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:37 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:35206
I20260806 11:44:57.647472 23982 tablet_replica.cc:330] stopping tablet replica
I20260806 11:44:57.647773 23982 raft_consensus.cc:2229] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:57.648042 23982 raft_consensus.cc:2258] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:57.649197 23982 ts_tablet_manager.cc:1905] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:57.652745 23982 ts_tablet_manager.cc:1918] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:57.652860 23982 log.cc:1196] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal/wals/8ee6d6b9a2dd47ed9cfd7cc25dc4473d
I20260806 11:44:57.653159 23982 ts_tablet_manager.cc:1939] T 8ee6d6b9a2dd47ed9cfd7cc25dc4473d P 25321ea43be8487fafa31f3b55e7dc67: Deleting consensus metadata
I20260806 11:44:57.654989 23682 catalog_manager.cc:4841] TS 25321ea43be8487fafa31f3b55e7dc67 (127.0.113.1:41043): tablet 8ee6d6b9a2dd47ed9cfd7cc25dc4473d (table test-table [id=19de7d4233b943f5b819de319c0da204]) successfully deleted
W20260806 11:44:57.734917 23974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:57.735155 23974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:57.735196 23974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:57.739135 23974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:57.739215 23974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:57.739313 23974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:44:57.744556 23974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:44159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=44221
--tserver_master_addrs=127.0.113.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:57.745512 23974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:57.746470 23974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:57.755954 23988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:57.755966 23989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:57.755970 23993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:57.756326 23974 server_base.cc:1034] running on GCE node
I20260806 11:44:57.756738 23974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:57.757377 23974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:57.758548 23974 hybrid_clock.cc:648] HybridClock initialized: now 1786016697758525 us; error 46 us; skew 500 ppm
Aug 06 11:44:57 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:57.761265 23974 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:44:57.762064 23974 webserver.cc:460] Webserver started at http://127.0.113.2:44221/ using document root <none> and password file <none>
I20260806 11:44:57.762566 23974 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:57.762634 23974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:57.766412 23974 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:44:57.768404 24003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:57.769637 23974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:44:57.769738 23974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "5b798d8f839e4933b6af83e607fab598"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "9c1fcfcf2ae9cc5a791d6018a4aaeaab"
server_key_iv: "f46ac3ba75786c0c347c0f789fd6b616"
server_key_version: "encryptionkey@0"
I20260806 11:44:57.770267 23974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:57.783982 23974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:57.786628 23974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:57.786775 23974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:57.787257 23974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:57.787992 23974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:57.788036 23974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:57.788113 23974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:57.788132 23974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:57.797519 23974 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:44159
I20260806 11:44:57.797549 24174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:44159 every 8 connection(s)
I20260806 11:44:57.798513 23974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:44:57.801417 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 23974
I20260806 11:44:57.802027 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:45501
--local_ip_for_outbound_sockets=127.0.113.3
--tserver_master_addrs=127.0.113.62:36885
--webserver_port=39347
--webserver_interface=127.0.113.3
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 11:44:57 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:57.814209 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:57.800437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:55893 (local address 127.0.113.62:36885)
0806 11:44:57.800577 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:57.800581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:57.801791 (+ 1210us) server_negotiation.cc:407] Connection header received
0806 11:44:57.803191 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:57.803194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:57.803243 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:57.803316 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:57.804799 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.805655 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.807349 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.807743 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.807881 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:57.810446 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:57.810450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:57.810453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:57.810488 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:57.811964 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:57.812581 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:57.812585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:57.812586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:57.812678 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:57.813049 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:57.813053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:57.813054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:57.813248 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:57.813360 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:57.813888 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 11:44:57.814029 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 11:44:57.814988 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:57.800807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:55893)
0806 11:44:57.801607 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:57.801663 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:57.802934 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:57.803499 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:57.803511 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:57.804056 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:57.804623 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:57.804638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.805805 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.805808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:57.807211 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:57.807224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:57.807926 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:57.807931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:57.808106 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:57.808742 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:57.808757 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:57.810262 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:57.812090 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:57.812095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:57.812109 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:57.812475 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:57.812771 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:57.812775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:57.812777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:57.812941 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:57.813367 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:57.813376 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:57.813620 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 11:44:57.813803 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 11:44:57.814124 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":112,"threads_started":1}
I20260806 11:44:57.816479 24175 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36885
I20260806 11:44:57.816725 24175 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:57.817211 24175 heartbeater.cc:511] Master 127.0.113.62:36885 requested a full tablet report, sending...
I20260806 11:44:57.818269 23704 ts_manager.cc:194] Registered new tserver with Master: 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159)
I20260806 11:44:57.818980 23704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:55893
W20260806 11:44:57.930787 24179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:44:57.931030 24179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:44:57.931069 24179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:44:57.934939 24179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:44:57.934990 24179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:44:57.935046 24179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:44:57.940542 24179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:45501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=39347
--tserver_master_addrs=127.0.113.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:44:57.941584 24179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:44:57.942662 24179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:44:57.952503 24185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:57.952589 24188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:44:57.952514 24184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:44:57.952950 24179 server_base.cc:1034] running on GCE node
I20260806 11:44:57.953328 24179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:44:57.953976 24179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:44:57.955184 24179 hybrid_clock.cc:648] HybridClock initialized: now 1786016697955175 us; error 32 us; skew 500 ppm
Aug 06 11:44:57 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:44:57.958112 24179 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:44:57.959018 24179 webserver.cc:460] Webserver started at http://127.0.113.3:39347/ using document root <none> and password file <none>
I20260806 11:44:57.959622 24179 fs_manager.cc:359] Metadata directory not provided
I20260806 11:44:57.959738 24179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:44:57.963611 24179 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:44:57.965852 24197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:57.967124 24179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:44:57.967268 24179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "b55f9dc9fd7a42838e66f008f94be29e"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "f1c2a1d0fd872134eeed2541ef68dcd8"
server_key_iv: "d8407bef3d55af692c4040b3b0878af4"
server_key_version: "encryptionkey@0"
I20260806 11:44:57.967813 24179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:44:58.003239 24179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:44:58.006631 24179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:44:58.006843 24179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:44:58.007472 24179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:44:58.008337 24179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:44:58.008410 24179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:58.008484 24179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:44:58.008525 24179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:44:58.017804 24179 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:45501
I20260806 11:44:58.017831 24369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:45501 every 8 connection(s)
I20260806 11:44:58.018827 24179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:44:58.020927 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 24179
Aug 06 11:44:58 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:44:58.031262 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.021646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.0.113.62:36885)
0806 11:44:58.021913 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:58.021917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:58.022116 (+ 199us) server_negotiation.cc:407] Connection header received
0806 11:44:58.022263 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:58.022266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:58.022319 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:58.022436 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:58.023362 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.024165 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.025279 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.025561 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.025672 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.027234 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:58.027237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:58.027240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:58.027268 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:58.029164 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:58.029669 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:58.029674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:58.029677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:58.029772 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:58.030122 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:58.030127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:58.030129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:58.030370 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:58.030519 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:58.030857 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 11:44:58.031038 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"spinlock_wait_cycles":7936,"thread_start_us":96,"threads_started":1}
I20260806 11:44:58.034416 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.020720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:45819 (local address 127.0.113.62:36885)
0806 11:44:58.020922 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:58.020927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:58.022136 (+ 1209us) server_negotiation.cc:407] Connection header received
0806 11:44:58.023162 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:58.023166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:58.023227 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:58.023317 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:58.024793 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.025448 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.026675 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.027064 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.027184 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.030302 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:58.030305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:58.030308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:58.030348 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:58.032425 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:58.032924 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:58.032928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:58.032930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:58.033025 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:58.033462 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:58.033465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:58.033466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:58.033639 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:58.033720 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:58.034188 (+ 468us) server_negotiation.cc:299] Negotiation successful
0806 11:44:58.034288 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 11:44:58.035293 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.021138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:45819)
0806 11:44:58.021967 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:58.022025 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:58.022961 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:58.023564 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:58.023574 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:58.024041 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:44:58.024654 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.024668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.025590 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.025596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.026548 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.026558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.027250 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.027254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.027483 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.028067 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:44:58.028092 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:44:58.030122 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:44:58.032559 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:58.032564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:58.032577 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:58.032827 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:58.033129 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:44:58.033135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:44:58.033139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:44:58.033344 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:44:58.033732 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:44:58.033737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:44:58.033973 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 11:44:58.034128 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 11:44:58.034397 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":157,"threads_started":1}
I20260806 11:44:58.036598 24370 heartbeater.cc:347] Connected to a master server at 127.0.113.62:36885
I20260806 11:44:58.036850 24370 heartbeater.cc:464] Registering TS with master...
I20260806 11:44:58.037371 24370 heartbeater.cc:511] Master 127.0.113.62:36885 requested a full tablet report, sending...
I20260806 11:44:58.038456 23704 ts_manager.cc:194] Registered new tserver with Master: b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501)
I20260806 11:44:58.039098 23704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:45819
I20260806 11:44:58.045413 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:44:58.049803 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.046056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.0.113.62:36885)
0806 11:44:58.046225 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:58.046229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:58.046410 (+ 181us) server_negotiation.cc:407] Connection header received
0806 11:44:58.046554 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:58.046557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:58.046657 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:58.046746 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:44:58.047130 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.047669 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.048596 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.048818 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.048919 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.049183 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:44:58.049326 (+ 143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 11:44:58.049655 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41252: token expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 11:44:58.049929 23971 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41252: token expired
W20260806 11:44:58.050149 23601 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.113.62:36885)
I20260806 11:44:58.057422 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.050570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.0.113.62:36885)
0806 11:44:58.050687 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:58.050690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:58.050719 (+ 29us) server_negotiation.cc:407] Connection header received
0806 11:44:58.050825 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:58.050827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:58.050886 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:58.050951 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:44:58.051543 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.052017 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.052840 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.053060 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.053138 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.054459 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:44:58.054462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:44:58.054464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:44:58.054485 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:44:58.055880 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:58.056268 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:58.056270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:58.056271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:58.056404 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:44:58.056666 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:44:58.056668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:44:58.056669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:44:58.056824 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:44:58.056947 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:58.057134 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 11:44:58.057222 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 11:44:58.059814 23704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:44:58.063104 23704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:44:58.070976 23874 tablet_service.cc:1467] Processing CreateTablet for tablet a7fe4e77d2de41f49fd6831ae434aea3 (DEFAULT_TABLE table=test-table [id=d000567db1b049c6823027bbcc7933b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:58.071626 23874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fe4e77d2de41f49fd6831ae434aea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:58.074712 23813 tablet_bootstrap.cc:492] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Bootstrap starting.
I20260806 11:44:58.075528 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.069768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:45501 (local address 127.0.0.1:35944)
0806 11:44:58.069901 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:58.069917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:58.070014 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:58.071194 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:58.071197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:58.071214 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:58.071308 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.071316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.072709 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.072712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.073895 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.073907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.075076 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.075080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.075183 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.075228 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 11:44:58.075316 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 11:44:58.075393 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 11:44:58.075631 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.070071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.0.113.3:45501)
0806 11:44:58.070619 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:58.070640 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:58.070659 (+ 19us) server_negotiation.cc:407] Connection header received
0806 11:44:58.070777 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:58.070782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:58.071009 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:58.071168 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:58.071476 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.072572 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.074083 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.074948 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.075061 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.075203 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:58.075363 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 11:44:58.075472 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":205,"threads_started":1}
I20260806 11:44:58.075665 23813 tablet_bootstrap.cc:654] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:58.075888 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.070065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:44159 (local address 127.0.0.1:49766)
0806 11:44:58.070351 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:58.070365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:58.070466 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:58.071364 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:58.071367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:58.071388 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:58.071501 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.071508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.072840 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.072844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.073783 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.073794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.075228 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.075233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.075353 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.075379 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:44:58.075452 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 11:44:58.075513 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20260806 11:44:58.076288 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.070528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.0.113.2:44159)
0806 11:44:58.070902 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:58.070914 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:58.070935 (+ 21us) server_negotiation.cc:407] Connection header received
0806 11:44:58.071003 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:58.071008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:58.071168 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:58.071646 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:58.071859 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.072706 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.073953 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.074515 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.074638 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.074788 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:58.075925 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0806 11:44:58.076156 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":147,"threads_started":1}
I20260806 11:44:58.077468 23813 tablet_bootstrap.cc:492] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: No bootstrap required, opened a new log
I20260806 11:44:58.077571 23813 ts_tablet_manager.cc:1397] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 11:44:58.077988 23813 raft_consensus.cc:348] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:44:58.078114 23813 raft_consensus.cc:374] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:58.078152 23813 raft_consensus.cc:729] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25321ea43be8487fafa31f3b55e7dc67, State: Initialized, Role: FOLLOWER
I20260806 11:44:58.078295 23813 consensus_queue.cc:260] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:44:58.079026 23813 ts_tablet_manager.cc:1428] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 11:44:58.078715 24272 tablet_service.cc:1467] Processing CreateTablet for tablet a7fe4e77d2de41f49fd6831ae434aea3 (DEFAULT_TABLE table=test-table [id=d000567db1b049c6823027bbcc7933b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:58.079138 24073 tablet_service.cc:1467] Processing CreateTablet for tablet a7fe4e77d2de41f49fd6831ae434aea3 (DEFAULT_TABLE table=test-table [id=d000567db1b049c6823027bbcc7933b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:44:58.079788 24272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fe4e77d2de41f49fd6831ae434aea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:58.080075 24073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fe4e77d2de41f49fd6831ae434aea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:44:58.087020 24389 tablet_bootstrap.cc:492] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: Bootstrap starting.
I20260806 11:44:58.087098 24390 tablet_bootstrap.cc:492] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: Bootstrap starting.
I20260806 11:44:58.089521 24389 tablet_bootstrap.cc:654] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:58.089540 24390 tablet_bootstrap.cc:654] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: Neither blocks nor log segments found. Creating new log.
I20260806 11:44:58.090348 24390 log.cc:824] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:58.090348 24389 log.cc:824] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: Log is configured to *not* fsync() on all Append() calls
I20260806 11:44:58.092103 24390 tablet_bootstrap.cc:492] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: No bootstrap required, opened a new log
I20260806 11:44:58.092348 24390 ts_tablet_manager.cc:1397] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 11:44:58.093248 24389 tablet_bootstrap.cc:492] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: No bootstrap required, opened a new log
I20260806 11:44:58.093433 24389 ts_tablet_manager.cc:1397] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 11:44:58.095511 24390 raft_consensus.cc:348] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:44:58.095811 24390 raft_consensus.cc:374] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:58.095872 24390 raft_consensus.cc:729] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b798d8f839e4933b6af83e607fab598, State: Initialized, Role: FOLLOWER
I20260806 11:44:58.096263 24390 consensus_queue.cc:260] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:44:58.096648 24389 raft_consensus.cc:348] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:44:58.096864 24389 raft_consensus.cc:374] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:44:58.096925 24389 raft_consensus.cc:729] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55f9dc9fd7a42838e66f008f94be29e, State: Initialized, Role: FOLLOWER
I20260806 11:44:58.096994 24175 heartbeater.cc:503] Master 127.0.113.62:36885 was elected leader, sending a full tablet report...
I20260806 11:44:58.097304 24390 ts_tablet_manager.cc:1428] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:44:58.097302 24389 consensus_queue.cc:260] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:44:58.097935 24370 heartbeater.cc:503] Master 127.0.113.62:36885 was elected leader, sending a full tablet report...
I20260806 11:44:58.098183 24389 ts_tablet_manager.cc:1428] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260806 11:44:58.100698 24394 raft_consensus.cc:482] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:44:58.100858 24394 raft_consensus.cc:504] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:44:58.102480 24394 leader_election.cc:288] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159), 25321ea43be8487fafa31f3b55e7dc67 (127.0.113.1:41043)
I20260806 11:44:58.105631 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.102065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:44159 (local address 127.0.113.3:49885)
0806 11:44:58.102231 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:58.102247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:58.102329 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:58.102707 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:58.102710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:58.102734 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:58.102917 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.102926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.103714 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.103719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.104574 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.104584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.105216 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.105219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.105321 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.105347 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:44:58.105433 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 11:44:58.105508 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"spinlock_wait_cycles":6400}
I20260806 11:44:58.105628 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.102157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:49885 (local address 127.0.113.2:44159)
0806 11:44:58.102276 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:58.102280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:58.102292 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:44:58.102457 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:58.102462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:58.102507 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:58.102583 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:58.103078 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.103604 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.104695 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.105101 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.105215 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.105272 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:58.105463 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 11:44:58.105528 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 11:44:58.105984 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.102246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:41043 (local address 127.0.113.3:43253)
0806 11:44:58.102630 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:58.102645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:44:58.102728 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:44:58.103134 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:44:58.103136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:44:58.103151 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:44:58.103239 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.103243 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.104037 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.104042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.104886 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:44:58.104895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.105585 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.105588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:44:58.105683 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.105704 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 11:44:58.105768 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 11:44:58.105831 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
I20260806 11:44:58.105973 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.102601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:43253 (local address 127.0.113.1:41043)
0806 11:44:58.102715 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:58.102720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:58.102737 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:44:58.102813 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:58.102816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:58.102886 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:58.103007 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:44:58.103359 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.103923 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.105023 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.105461 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.105581 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.105733 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:58.105777 (+ 44us) server_negotiation.cc:299] Negotiation successful
0806 11:44:58.105841 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 11:44:58.106596 24112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe4e77d2de41f49fd6831ae434aea3" candidate_uuid: "b55f9dc9fd7a42838e66f008f94be29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b798d8f839e4933b6af83e607fab598" is_pre_election: true
I20260806 11:44:58.106602 23902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe4e77d2de41f49fd6831ae434aea3" candidate_uuid: "b55f9dc9fd7a42838e66f008f94be29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25321ea43be8487fafa31f3b55e7dc67" is_pre_election: true
I20260806 11:44:58.106881 24112 raft_consensus.cc:2454] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b55f9dc9fd7a42838e66f008f94be29e in term 0.
I20260806 11:44:58.106896 23902 raft_consensus.cc:2454] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b55f9dc9fd7a42838e66f008f94be29e in term 0.
I20260806 11:44:58.107340 24204 leader_election.cc:302] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25321ea43be8487fafa31f3b55e7dc67, b55f9dc9fd7a42838e66f008f94be29e; no voters:
W20260806 11:44:58.107520 23969 tablet.cc:2369] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:44:58.107650 24394 raft_consensus.cc:2781] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:44:58.107762 24394 raft_consensus.cc:482] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:44:58.107823 24394 raft_consensus.cc:3037] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:58.108861 24394 raft_consensus.cc:504] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:44:58.109236 24394 leader_election.cc:288] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [CANDIDATE]: Term 1 election: Requested vote from peers 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159), 25321ea43be8487fafa31f3b55e7dc67 (127.0.113.1:41043)
I20260806 11:44:58.109555 24112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe4e77d2de41f49fd6831ae434aea3" candidate_uuid: "b55f9dc9fd7a42838e66f008f94be29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b798d8f839e4933b6af83e607fab598"
I20260806 11:44:58.109573 23902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe4e77d2de41f49fd6831ae434aea3" candidate_uuid: "b55f9dc9fd7a42838e66f008f94be29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25321ea43be8487fafa31f3b55e7dc67"
I20260806 11:44:58.109678 24112 raft_consensus.cc:3037] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:58.109678 23902 raft_consensus.cc:3037] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:44:58.110934 23902 raft_consensus.cc:2454] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b55f9dc9fd7a42838e66f008f94be29e in term 1.
I20260806 11:44:58.110934 24112 raft_consensus.cc:2454] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b55f9dc9fd7a42838e66f008f94be29e in term 1.
I20260806 11:44:58.111264 24200 leader_election.cc:302] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b798d8f839e4933b6af83e607fab598, b55f9dc9fd7a42838e66f008f94be29e; no voters:
I20260806 11:44:58.111439 24394 raft_consensus.cc:2781] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:44:58.111657 24394 raft_consensus.cc:686] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 1 LEADER]: Becoming Leader. State: Replica: b55f9dc9fd7a42838e66f008f94be29e, State: Running, Role: LEADER
I20260806 11:44:58.111940 24394 consensus_queue.cc:237] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:44:58.114946 23703 catalog_manager.cc:5495] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e reported cstate change: term changed from 0 to 1, leader changed from <none> to b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3). New cstate: current_term: 1 leader_uuid: "b55f9dc9fd7a42838e66f008f94be29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } health_report { overall_health: UNKNOWN } } }
I20260806 11:44:58.129628 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.126089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.0.113.3:45501)
0806 11:44:58.126234 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 11:44:58.126238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:44:58.126252 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:44:58.126430 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:44:58.126433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:44:58.126482 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:44:58.126590 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:44:58.126862 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.127339 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.128143 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:44:58.128419 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:44:58.128527 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:44:58.128833 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:44:58.129199 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:44:58.129330 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 11:44:58.129405 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 11:44:58.129457 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 11:44:58.135555 23902 raft_consensus.cc:1261] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Refusing update from remote peer b55f9dc9fd7a42838e66f008f94be29e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:58.135567 24112 raft_consensus.cc:1261] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [term 1 FOLLOWER]: Refusing update from remote peer b55f9dc9fd7a42838e66f008f94be29e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:44:58.136245 24394 consensus_queue.cc:1035] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:58.136488 24398 consensus_queue.cc:1035] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:44:58.142094 24401 mvcc.cc:205] Tried to move back new op lower bound from 7315524395556544512 to 7315524395469520896. Current Snapshot: MvccSnapshot[applied={T|T < 7315524395556544512}]
I20260806 11:44:58.142493 24402 mvcc.cc:205] Tried to move back new op lower bound from 7315524395556544512 to 7315524395469520896. Current Snapshot: MvccSnapshot[applied={T|T < 7315524395556544512}]
I20260806 11:44:58.152436 23703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:58.152616 23703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:44:58.155282 23703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Sending DeleteTablet for 3 replicas of tablet a7fe4e77d2de41f49fd6831ae434aea3
I20260806 11:44:58.155970 24272 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fe4e77d2de41f49fd6831ae434aea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:58 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:35944
I20260806 11:44:58.156102 23874 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fe4e77d2de41f49fd6831ae434aea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:58 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:35206
I20260806 11:44:58.156121 24073 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fe4e77d2de41f49fd6831ae434aea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:44:58 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:49766
I20260806 11:44:58.156497 24416 tablet_replica.cc:330] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: stopping tablet replica
I20260806 11:44:58.156517 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 23750
I20260806 11:44:58.156649 24417 tablet_replica.cc:330] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: stopping tablet replica
I20260806 11:44:58.156925 24417 raft_consensus.cc:2229] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:44:58.157249 24417 raft_consensus.cc:2258] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:58.161132 24417 ts_tablet_manager.cc:1905] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:58.164142 24417 ts_tablet_manager.cc:1918] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:58.164234 24417 log.cc:1196] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/a7fe4e77d2de41f49fd6831ae434aea3
I20260806 11:44:58.164557 24417 ts_tablet_manager.cc:1939] T a7fe4e77d2de41f49fd6831ae434aea3 P 5b798d8f839e4933b6af83e607fab598: Deleting consensus metadata
I20260806 11:44:58.165714 23676 catalog_manager.cc:4841] TS 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159): tablet a7fe4e77d2de41f49fd6831ae434aea3 (table test-table [id=d000567db1b049c6823027bbcc7933b9]) successfully deleted
I20260806 11:44:58.166198 24416 raft_consensus.cc:2229] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:44:58.166543 24416 raft_consensus.cc:2258] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:44:58.168167 24416 ts_tablet_manager.cc:1905] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:44:58.171212 24416 ts_tablet_manager.cc:1918] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:44:58.171324 24416 log.cc:1196] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/a7fe4e77d2de41f49fd6831ae434aea3
I20260806 11:44:58.171674 24416 ts_tablet_manager.cc:1939] T a7fe4e77d2de41f49fd6831ae434aea3 P b55f9dc9fd7a42838e66f008f94be29e: Deleting consensus metadata
I20260806 11:44:58.172735 23676 catalog_manager.cc:4841] TS b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501): tablet a7fe4e77d2de41f49fd6831ae434aea3 (table test-table [id=d000567db1b049c6823027bbcc7933b9]) successfully deleted
W20260806 11:44:58.178608 23682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:41043 (error 108)
I20260806 11:44:58.178758 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 23974
I20260806 11:44:58.187824 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:44:58.187534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:41043 (local address 127.0.0.1:35216)
0806 11:44:58.187675 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 11:44:58.187750 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:41043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 11:44:58.188054 23682 catalog_manager.cc:4584] TS 25321ea43be8487fafa31f3b55e7dc67 (127.0.113.1:41043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7fe4e77d2de41f49fd6831ae434aea3: Network error: Client connection negotiation failed: client connection to 127.0.113.1:41043: connect: Connection refused (error 111)
I20260806 11:44:58.193812 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 24179
I20260806 11:44:58.210000 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 23650
I20260806 11:45:18.221402 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36885
--webserver_interface=127.0.113.62
--webserver_port=34917
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:18.339206 24423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:18.339496 24423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:18.339535 24423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:18.343526 24423 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 11:45:18.343575 24423 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 11:45:18.343590 24423 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 11:45:18.343627 24423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:18.343664 24423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:18.343703 24423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:18.343744 24423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:18.343789 24423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:18.348981 24423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=34917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:18.349934 24423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:18.350919 24423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:18.359474 24431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:18.359648 24435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:18.359515 24423 server_base.cc:1034] running on GCE node
W20260806 11:45:18.359532 24433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:18.360431 24423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:18.361460 24423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:18.362627 24423 hybrid_clock.cc:648] HybridClock initialized: now 1786016718362582 us; error 54 us; skew 500 ppm
Aug 06 11:45:18 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:18.372648 24423 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:45:18.373577 24423 webserver.cc:460] Webserver started at http://127.0.113.62:34917/ using document root <none> and password file <none>
I20260806 11:45:18.374162 24423 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:18.374228 24423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:18.378146 24423 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:45:18.380443 24448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:18.381664 24423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:45:18.381811 24423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "f367b8bcac544826a0be18e1bb47c40f"
format_stamp: "Formatted at 2026-08-06 11:44:35 on dist-test-slave-45dx"
server_key: "c32af28ed485c4abc26442bff9bad70b"
server_key_iv: "b9d3cd6b805e5a18d1e69a4d589ff025"
server_key_version: "encryptionkey@0"
I20260806 11:45:18.382365 24423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:18.408475 24423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:18.411976 24423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:18.412194 24423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:18.419737 24423 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:36885
I20260806 11:45:18.420082 24526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:36885 every 8 connection(s)
I20260806 11:45:18.420933 24423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:18.425239 24531 sys_catalog.cc:263] Verifying existing consensus state
I20260806 11:45:18.426883 24531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Bootstrap starting.
I20260806 11:45:18.429651 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 24423
I20260806 11:45:18.430096 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:41043
--local_ip_for_outbound_sockets=127.0.113.1
--tserver_master_addrs=127.0.113.62:36885
--webserver_port=34249
--webserver_interface=127.0.113.1
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:45:18.446518 24531 log.cc:824] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:18.465387 24531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:45:18.465911 24531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Bootstrap complete.
I20260806 11:45:18.473034 24531 raft_consensus.cc:348] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:45:18.473609 24531 raft_consensus.cc:729] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f367b8bcac544826a0be18e1bb47c40f, State: Initialized, Role: FOLLOWER
I20260806 11:45:18.474181 24531 consensus_queue.cc:260] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:45:18.474355 24531 raft_consensus.cc:388] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:18.474438 24531 raft_consensus.cc:482] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:18.474560 24531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 2 FOLLOWER]: Advancing to term 3
I20260806 11:45:18.477252 24531 raft_consensus.cc:504] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:45:18.477630 24531 leader_election.cc:302] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f367b8bcac544826a0be18e1bb47c40f; no voters:
I20260806 11:45:18.478070 24531 leader_election.cc:288] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 11:45:18.478159 24535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 3 FOLLOWER]: Leader election won for term 3
I20260806 11:45:18.478370 24535 raft_consensus.cc:686] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [term 3 LEADER]: Becoming Leader. State: Replica: f367b8bcac544826a0be18e1bb47c40f, State: Running, Role: LEADER
I20260806 11:45:18.478690 24535 consensus_queue.cc:237] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } }
I20260806 11:45:18.479400 24531 sys_catalog.cc:564] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:18.479713 24535 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f367b8bcac544826a0be18e1bb47c40f. Latest consensus state: current_term: 3 leader_uuid: "f367b8bcac544826a0be18e1bb47c40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } } }
I20260806 11:45:18.479838 24535 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:18.480211 24542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:18.480360 24537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f367b8bcac544826a0be18e1bb47c40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367b8bcac544826a0be18e1bb47c40f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36885 } } }
I20260806 11:45:18.480540 24537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:18.485698 24542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:18.486642 24542 catalog_manager.cc:1259] Loaded cluster ID: c22d92cc466f4e019c66acfe64023c05
I20260806 11:45:18.486711 24542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:18.489723 24542 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:18.490602 24542 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Loaded TSK: 0
I20260806 11:45:18.490695 24542 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f367b8bcac544826a0be18e1bb47c40f: Loaded TSK: 1
F20260806 11:45:18.491616 24542 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb390c6b9ad google::LogMessage::Fail() at ??:0
@ 0x7fb390c6e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb390c6b4de google::LogMessage::Flush() at ??:0
@ 0x7fb390c6ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb39392f18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb393927f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb39b5d07bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb39b5aa99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb39b5ab178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fb39b5ecdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb39b60b82b std::function<>::operator()() at ??:0
@ 0x7fb39b5aab4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb39b5abf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb39b5a8579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb39b5ec640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb392ebd3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb392ebdc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb392ebf3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb392eae512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb3944f26db start_thread at ??:0
@ 0x7fb38f76961f clone at ??:0
W20260806 11:45:18.582305 24533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:18.582531 24533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:18.582574 24533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:18.587018 24533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:18.587064 24533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:18.587119 24533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:45:18.592799 24533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:41043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=34249
--tserver_master_addrs=127.0.113.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:18.593822 24533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:18.594796 24533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:18.604574 24600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:18.604704 24602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:18.604574 24604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:18.605926 24533 server_base.cc:1034] running on GCE node
I20260806 11:45:18.606374 24533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:18.607079 24533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:18.608271 24533 hybrid_clock.cc:648] HybridClock initialized: now 1786016718608246 us; error 57 us; skew 500 ppm
Aug 06 11:45:18 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:18.611213 24533 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:45:18.612495 24533 webserver.cc:460] Webserver started at http://127.0.113.1:34249/ using document root <none> and password file <none>
I20260806 11:45:18.613094 24533 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:18.613157 24533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:18.617368 24533 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:45:18.619701 24615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:18.621145 24533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 11:45:18.621290 24533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "25321ea43be8487fafa31f3b55e7dc67"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "0f8995d8bc1f26bd4f4b2335ae410ece"
server_key_iv: "8a98c86c4d3f133ceb58391550445f37"
server_key_version: "encryptionkey@0"
I20260806 11:45:18.621857 24533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:18.646781 24533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:18.650241 24533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:18.650456 24533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:18.651101 24533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:18.652264 24624 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 11:45:18.657864 24533 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 11:45:18.657950 24533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 11:45:18.658057 24533 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 11:45:18.659701 24533 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 11:45:18.659772 24533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 11:45:18.660219 24624 tablet_bootstrap.cc:492] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Bootstrap starting.
I20260806 11:45:18.670651 24533 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:41043
I20260806 11:45:18.670914 24772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:41043 every 8 connection(s)
I20260806 11:45:18.671685 24533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:45:18.678246 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 24533
I20260806 11:45:18.678696 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:44159
--local_ip_for_outbound_sockets=127.0.113.2
--tserver_master_addrs=127.0.113.62:36885
--webserver_port=44221
--webserver_interface=127.0.113.2
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:45:18.682560 24624 log.cc:824] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:18.691963 24624 tablet_bootstrap.cc:492] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 11:45:18.692471 24624 tablet_bootstrap.cc:492] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Bootstrap complete.
I20260806 11:45:18.694178 24624 ts_tablet_manager.cc:1397] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.011s
I20260806 11:45:18.697531 24624 raft_consensus.cc:348] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:45:18.698096 24624 raft_consensus.cc:729] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25321ea43be8487fafa31f3b55e7dc67, State: Initialized, Role: FOLLOWER
I20260806 11:45:18.698624 24624 consensus_queue.cc:260] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:45:18.699716 24624 ts_tablet_manager.cc:1428] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260806 11:45:18.748525 24775 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:45:18.673713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:52417)
0806 11:45:18.674195 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:18.674249 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:18.675064 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:18.747439 (+ 72375us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":153,"threads_started":1}
W20260806 11:45:18.749150 24773 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:36885 (0 consecutive failures): Network error: Failed to ping master at 127.0.113.62:36885: Client connection negotiation failed: client connection to 127.0.113.62:36885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 11:45:18.749917 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:18.749609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:57549)
0806 11:45:18.749771 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:18.749851 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 11:45:18.750592 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:18.750403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:49759)
0806 11:45:18.750506 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:18.750547 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 11:45:18.750841 24773 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 11:45:18.794430 24779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:18.794716 24779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:18.794788 24779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:18.798763 24779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:18.798858 24779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:18.798955 24779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:45:18.804471 24779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:44159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=44221
--tserver_master_addrs=127.0.113.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:18.805538 24779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:18.806563 24779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:18.816205 24793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:18.816346 24779 server_base.cc:1034] running on GCE node
W20260806 11:45:18.816205 24791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:18.816526 24795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:18.816938 24779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:18.817561 24779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:18.818720 24779 hybrid_clock.cc:648] HybridClock initialized: now 1786016718818689 us; error 41 us; skew 500 ppm
Aug 06 11:45:18 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:18.821725 24779 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:45:18.822712 24779 webserver.cc:460] Webserver started at http://127.0.113.2:44221/ using document root <none> and password file <none>
I20260806 11:45:18.823267 24779 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:18.823380 24779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:18.827421 24779 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 11:45:18.829476 24811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:18.830627 24779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 11:45:18.830760 24779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "5b798d8f839e4933b6af83e607fab598"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "9c1fcfcf2ae9cc5a791d6018a4aaeaab"
server_key_iv: "f46ac3ba75786c0c347c0f789fd6b616"
server_key_version: "encryptionkey@0"
I20260806 11:45:18.831300 24779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:18.855356 24779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:18.858913 24779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:18.859143 24779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:18.859771 24779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:18.860659 24779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:18.860731 24779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:18.860806 24779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:18.860849 24779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:18.870568 24779 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:44159
I20260806 11:45:18.870595 24978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:44159 every 8 connection(s)
I20260806 11:45:18.871631 24779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:45:18.875316 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:18.873823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:40017)
0806 11:45:18.874521 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:18.874632 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":115,"threads_started":1}
W20260806 11:45:18.875804 24979 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:36885 (0 consecutive failures): Network error: Failed to ping master at 127.0.113.62:36885: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
I20260806 11:45:18.876438 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:18.876206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:36839)
0806 11:45:18.876322 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:18.876371 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 11:45:18.876757 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 24779
I20260806 11:45:18.877137 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:18.876916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:57667)
0806 11:45:18.877014 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:18.877089 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 11:45:18.877259 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:45501
--local_ip_for_outbound_sockets=127.0.113.3
--tserver_master_addrs=127.0.113.62:36885
--webserver_port=39347
--webserver_interface=127.0.113.3
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:18.877358 24979 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 11:45:18.990618 24982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:18.990885 24982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:18.990928 24982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:18.995174 24982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:18.995245 24982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:18.995337 24982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:45:19.000763 24982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:45501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=39347
--tserver_master_addrs=127.0.113.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:19.001876 24982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:19.002905 24982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:19.012535 24989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:19.012527 24990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:19.012529 24997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:19.012964 24982 server_base.cc:1034] running on GCE node
I20260806 11:45:19.013376 24982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:19.014000 24982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:19.015189 24982 hybrid_clock.cc:648] HybridClock initialized: now 1786016719015168 us; error 32 us; skew 500 ppm
Aug 06 11:45:19 dist-test-slave-45dx krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:19.018061 24982 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:45:19.018958 24982 webserver.cc:460] Webserver started at http://127.0.113.3:39347/ using document root <none> and password file <none>
I20260806 11:45:19.019465 24982 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:19.019527 24982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:19.023232 24982 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:19.025363 25006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:19.026628 24982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:19.026721 24982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "b55f9dc9fd7a42838e66f008f94be29e"
format_stamp: "Formatted at 2026-08-06 11:44:36 on dist-test-slave-45dx"
server_key: "f1c2a1d0fd872134eeed2541ef68dcd8"
server_key_iv: "d8407bef3d55af692c4040b3b0878af4"
server_key_version: "encryptionkey@0"
I20260806 11:45:19.027225 24982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:19.043462 24982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:19.046938 24982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:19.047112 24982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:19.047672 24982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:19.048548 24982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:19.048599 24982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:19.048641 24982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:19.048656 24982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:19.058200 24982 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:45501
I20260806 11:45:19.058224 25183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:45501 every 8 connection(s)
I20260806 11:45:19.059271 24982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:45:19.062955 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:19.061461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:58879)
0806 11:45:19.062159 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:19.062270 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":127,"threads_started":1}
W20260806 11:45:19.063390 25184 heartbeater.cc:644] Failed to heartbeat to 127.0.113.62:36885 (0 consecutive failures): Network error: Failed to ping master at 127.0.113.62:36885: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
I20260806 11:45:19.064021 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:19.063796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:43483)
0806 11:45:19.063911 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:19.063959 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 11:45:19.064662 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:19.064452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:42641)
0806 11:45:19.064559 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:19.064604 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 11:45:19.064690 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 24982
W20260806 11:45:19.064899 25184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 11:45:19.751817 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:19.751388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:40477)
0806 11:45:19.751635 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:19.751688 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260806 11:45:19.878257 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:19.877865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:48875)
0806 11:45:19.878103 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:19.878156 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260806 11:45:20.065828 25195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:20.065421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:51743)
0806 11:45:20.065671 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:20.065728 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260806 11:45:20.243110 25197 raft_consensus.cc:482] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:20.243304 25197 raft_consensus.cc:504] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:45:20.244380 25197 leader_election.cc:288] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501), 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159)
Aug 06 11:45:20 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.3@KRBTEST.COM
Aug 06 11:45:20 dist-test-slave-45dx krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.2@KRBTEST.COM
I20260806 11:45:20.258013 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:20.244901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:44159 (local address 127.0.113.1:43543)
0806 11:45:20.245201 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:20.245214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:20.245307 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:20.247010 (+ 1703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:20.247013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:20.247296 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:20.247762 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:20.247776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:20.249140 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:20.249145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:20.249974 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:20.249983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:20.250664 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:20.250668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:20.250885 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:20.251631 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:20.251642 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:20.253608 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:20.255968 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:20.255971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:20.255973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:20.256221 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:20.256588 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:20.256591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:20.256593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:20.256683 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:20.257329 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:20.257332 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:20.257469 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 11:45:20.257581 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 11:45:20.257833 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":134,"spinlock_wait_cycles":16128,"thread_start_us":98,"threads_started":1}
I20260806 11:45:20.258212 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:20.244607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:45501 (local address 127.0.113.1:51039)
0806 11:45:20.244751 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:20.244767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:20.244887 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:20.246702 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:20.246708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:20.247248 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:20.247748 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:20.247776 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:20.249267 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:20.249270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:20.249979 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:20.249988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:20.250671 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:20.250675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:20.250884 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:20.251577 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:20.251593 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:20.253242 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:20.255351 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:20.255356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:20.255370 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:20.255718 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:20.256117 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:20.256119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:20.256121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:20.256247 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:20.257026 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:20.257032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:20.257287 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 11:45:20.257492 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 11:45:20.257976 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":257,"spinlock_wait_cycles":12160}
I20260806 11:45:20.258339 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:20.244777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:51039 (local address 127.0.113.3:45501)
0806 11:45:20.245160 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:20.245189 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:20.245242 (+ 53us) server_negotiation.cc:407] Connection header received
0806 11:45:20.245999 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:20.246006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:20.246342 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:20.246652 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:20.248093 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:20.249111 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:20.250106 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:20.250546 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:20.250715 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:20.253489 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:20.253496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:20.253512 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:20.253554 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:20.255229 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:20.255839 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:20.255843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:20.255847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:20.256011 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:20.256403 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:20.256405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:20.256406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:20.256914 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:20.257014 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:20.257867 (+ 853us) server_negotiation.cc:299] Negotiation successful
0806 11:45:20.258175 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260806 11:45:20.258790 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:20.245073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:43543 (local address 127.0.113.2:44159)
0806 11:45:20.245401 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:20.245421 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:20.245452 (+ 31us) server_negotiation.cc:407] Connection header received
0806 11:45:20.246087 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:20.246092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:20.246380 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:20.246692 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:20.247977 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:20.248998 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:20.250098 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:20.250543 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:20.250713 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:20.253787 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:20.253791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:20.253802 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:20.253828 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:20.255864 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:20.256344 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:20.256352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:20.256374 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:20.256487 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:20.256787 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:20.256789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:20.256790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:20.257212 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:20.257340 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:20.258294 (+ 954us) server_negotiation.cc:299] Negotiation successful
0806 11:45:20.258608 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":124,"threads_started":1}
W20260806 11:45:20.261019 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
W20260806 11:45:20.261323 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
I20260806 11:45:20.261423 24617 leader_election.cc:302] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25321ea43be8487fafa31f3b55e7dc67; no voters: 5b798d8f839e4933b6af83e607fab598, b55f9dc9fd7a42838e66f008f94be29e
I20260806 11:45:20.261740 25197 raft_consensus.cc:2726] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:45:20.753013 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:20.752721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:58281)
0806 11:45:20.752861 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:20.752926 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 11:45:20.879416 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:20.878988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:55691)
0806 11:45:20.879234 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:20.879306 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260806 11:45:21.066973 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:21.066535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:60003)
0806 11:45:21.066835 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:21.066888 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
I20260806 11:45:21.754274 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:21.753825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:40119)
0806 11:45:21.754105 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:21.754171 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260806 11:45:21.880692 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:21.880208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:60551)
0806 11:45:21.880527 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:21.880584 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":154,"threads_started":1}
I20260806 11:45:22.068377 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:22.067826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:35157)
0806 11:45:22.068135 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:22.068195 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":112,"threads_started":1}
I20260806 11:45:22.443159 25213 raft_consensus.cc:482] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:22.443359 25213 raft_consensus.cc:504] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:45:22.443915 25213 leader_election.cc:288] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501), 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159)
W20260806 11:45:22.444885 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
W20260806 11:45:22.445057 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
I20260806 11:45:22.445202 24617 leader_election.cc:302] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25321ea43be8487fafa31f3b55e7dc67; no voters: 5b798d8f839e4933b6af83e607fab598, b55f9dc9fd7a42838e66f008f94be29e
I20260806 11:45:22.445371 25213 raft_consensus.cc:2726] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:45:22.755463 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:22.755023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:55743)
0806 11:45:22.755300 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:22.755359 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260806 11:45:22.881937 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:22.881430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:51229)
0806 11:45:22.881736 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:22.881838 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260806 11:45:23.069758 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:23.069339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:43109)
0806 11:45:23.069601 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:23.069662 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260806 11:45:23.756930 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:23.756467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:49625)
0806 11:45:23.756765 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:23.756824 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":76,"threads_started":1}
I20260806 11:45:23.883328 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:23.882888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:51753)
0806 11:45:23.883166 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:23.883228 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260806 11:45:24.071081 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:24.070656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:39991)
0806 11:45:24.070924 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:24.070983 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260806 11:45:24.758287 25221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:24.757839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:56609)
0806 11:45:24.758122 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:24.758182 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260806 11:45:24.810024 25224 raft_consensus.cc:482] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:24.810274 25224 raft_consensus.cc:504] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:45:24.810758 25224 leader_election.cc:288] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501), 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159)
W20260806 11:45:24.811661 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
W20260806 11:45:24.811803 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
I20260806 11:45:24.811869 24617 leader_election.cc:302] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25321ea43be8487fafa31f3b55e7dc67; no voters: 5b798d8f839e4933b6af83e607fab598, b55f9dc9fd7a42838e66f008f94be29e
I20260806 11:45:24.812054 25224 raft_consensus.cc:2726] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:45:24.884709 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:24.884169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:47617)
0806 11:45:24.884505 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:24.884586 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":119,"threads_started":1}
I20260806 11:45:25.072258 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:25.071844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:48301)
0806 11:45:25.072082 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:25.072137 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260806 11:45:25.759764 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:25.759280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:60855)
0806 11:45:25.759580 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:25.759646 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260806 11:45:25.885895 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:25.885466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:51471)
0806 11:45:25.885744 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:25.885799 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
I20260806 11:45:26.073622 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:26.073131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:53249)
0806 11:45:26.073441 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:26.073519 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":78,"threads_started":1}
I20260806 11:45:26.761246 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:26.760803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:43807)
0806 11:45:26.761084 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:26.761144 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
I20260806 11:45:26.887075 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:26.886649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:56181)
0806 11:45:26.886915 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:26.886976 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260806 11:45:27.074806 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:27.074394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:43853)
0806 11:45:27.074652 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:27.074712 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260806 11:45:27.762472 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:27.762046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:41311)
0806 11:45:27.762307 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:27.762374 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260806 11:45:27.888293 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:27.887834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:44991)
0806 11:45:27.888127 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:27.888183 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
I20260806 11:45:28.076053 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:28.075592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:56631)
0806 11:45:28.075891 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:28.075952 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":98,"threads_started":1}
I20260806 11:45:28.763783 25242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:28.763372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:49355)
0806 11:45:28.763626 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:28.763683 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260806 11:45:28.889662 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:28.889212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:41657)
0806 11:45:28.889495 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:28.889561 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260806 11:45:29.077584 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:29.077067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:35001)
0806 11:45:29.077405 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:29.077477 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":82,"threads_started":1}
I20260806 11:45:29.410352 25247 raft_consensus.cc:482] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:29.410568 25247 raft_consensus.cc:504] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:45:29.411055 25247 leader_election.cc:288] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501), 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159)
W20260806 11:45:29.412037 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
W20260806 11:45:29.412221 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
I20260806 11:45:29.412297 24617 leader_election.cc:302] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25321ea43be8487fafa31f3b55e7dc67; no voters: 5b798d8f839e4933b6af83e607fab598, b55f9dc9fd7a42838e66f008f94be29e
I20260806 11:45:29.412483 25247 raft_consensus.cc:2726] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:45:29.765178 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:29.764700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:47337)
0806 11:45:29.765003 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:29.765072 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":82,"threads_started":1}
I20260806 11:45:29.891037 25250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:29.890559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:37063)
0806 11:45:29.890857 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:29.890929 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
I20260806 11:45:30.079066 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:30.078573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:55491)
0806 11:45:30.078897 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:30.078963 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
I20260806 11:45:30.766417 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:30.765998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:51435)
0806 11:45:30.766265 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:30.766318 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260806 11:45:30.892500 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:30.891980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:59243)
0806 11:45:30.892253 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:30.892386 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260806 11:45:31.080672 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:31.080063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:52821)
0806 11:45:31.080445 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:31.080532 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":132,"threads_started":1}
I20260806 11:45:31.767814 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:31.767326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:44027)
0806 11:45:31.767614 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:31.767680 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260806 11:45:31.893640 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:31.893231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:37113)
0806 11:45:31.893484 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:31.893539 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260806 11:45:32.081910 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:32.081475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:38535)
0806 11:45:32.081737 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:32.081811 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260806 11:45:32.769178 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:32.768681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:55389)
0806 11:45:32.769007 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:32.769073 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260806 11:45:32.895006 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:32.894522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:51659)
0806 11:45:32.894817 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:32.894898 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260806 11:45:33.074980 25260 raft_consensus.cc:482] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:33.075126 25260 raft_consensus.cc:504] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f9dc9fd7a42838e66f008f94be29e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 45501 } } peers { permanent_uuid: "5b798d8f839e4933b6af83e607fab598" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 44159 } } peers { permanent_uuid: "25321ea43be8487fafa31f3b55e7dc67" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41043 } }
I20260806 11:45:33.075629 25260 leader_election.cc:288] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501), 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159)
W20260806 11:45:33.076555 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b55f9dc9fd7a42838e66f008f94be29e (127.0.113.3:45501): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
W20260806 11:45:33.076745 24617 leader_election.cc:341] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b798d8f839e4933b6af83e607fab598 (127.0.113.2:44159): Not found: Tablet not found: a7fe4e77d2de41f49fd6831ae434aea3
I20260806 11:45:33.076812 24617 leader_election.cc:302] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25321ea43be8487fafa31f3b55e7dc67; no voters: 5b798d8f839e4933b6af83e607fab598, b55f9dc9fd7a42838e66f008f94be29e
I20260806 11:45:33.077028 25260 raft_consensus.cc:2726] T a7fe4e77d2de41f49fd6831ae434aea3 P 25321ea43be8487fafa31f3b55e7dc67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 11:45:33.083084 25263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:33.082659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:40053)
0806 11:45:33.082928 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:33.082982 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260806 11:45:33.770766 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:33.770309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.1:53807)
0806 11:45:33.770597 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:33.770660 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
I20260806 11:45:33.896494 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:33.895983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.2:50717)
0806 11:45:33.896269 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:33.896379 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
I20260806 11:45:34.069270 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 24533
I20260806 11:45:34.082753 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 24779
I20260806 11:45:34.086941 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:34.085857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36885 (local address 127.0.113.3:54583)
0806 11:45:34.086744 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:34.086811 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.62:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":607,"threads_started":1}
I20260806 11:45:34.092993 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 24982
2026-08-06T11:45:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59635 ms)
[----------] 30 tests from SecurityITest (245255 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:34 dist-test-slave-45dx krb5kdc[25272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:34 dist-test-slave-45dx krb5kdc[25272](info): set up 2 sockets
Aug 06 11:45:34 dist-test-slave-45dx krb5kdc[25272](info): commencing operation
krb5kdc: starting...
W20260806 11:45:35.350240 452 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:35 dist-test-slave-45dx krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:45:35.479532 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:37551
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:37551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:35.593530 25289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:35.593806 25289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:35.593894 25289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:35.597921 25289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:35.597978 25289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:35.597997 25289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:35.598047 25289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:35.598079 25289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:35.603118 25289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:37551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:37551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:35.604077 25289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:35.605126 25289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:35.613338 25302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:35.613351 25305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:35.613344 25299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:35.613695 25289 server_base.cc:1034] running on GCE node
I20260806 11:45:35.614259 25289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:35.615247 25289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:35.616402 25289 hybrid_clock.cc:648] HybridClock initialized: now 1786016735616383 us; error 31 us; skew 500 ppm
Aug 06 11:45:35 dist-test-slave-45dx krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:35.619088 25289 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:45:35.619937 25289 webserver.cc:460] Webserver started at http://127.0.113.62:42259/ using document root <none> and password file <none>
I20260806 11:45:35.620561 25289 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:35.620625 25289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:35.620862 25289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:35.622864 25289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "6bf6319b1c814a1f8d3b6410e8c9c468"
format_stamp: "Formatted at 2026-08-06 11:45:35 on dist-test-slave-45dx"
server_key: "fac194b4dbb429f0ac076ace66225ad8"
server_key_iv: "b865b50d7219f842eb87450b9f4b9fba"
server_key_version: "encryptionkey@0"
I20260806 11:45:35.623454 25289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "6bf6319b1c814a1f8d3b6410e8c9c468"
format_stamp: "Formatted at 2026-08-06 11:45:35 on dist-test-slave-45dx"
server_key: "fac194b4dbb429f0ac076ace66225ad8"
server_key_iv: "b865b50d7219f842eb87450b9f4b9fba"
server_key_version: "encryptionkey@0"
I20260806 11:45:35.627130 25289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:45:35.629560 25318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:35.630802 25289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:45:35.630932 25289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "6bf6319b1c814a1f8d3b6410e8c9c468"
format_stamp: "Formatted at 2026-08-06 11:45:35 on dist-test-slave-45dx"
server_key: "fac194b4dbb429f0ac076ace66225ad8"
server_key_iv: "b865b50d7219f842eb87450b9f4b9fba"
server_key_version: "encryptionkey@0"
I20260806 11:45:35.631033 25289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:35.645210 25289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:35.647997 25289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:35.648206 25289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:35.655268 25289 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:37551
I20260806 11:45:35.655292 25386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:37551 every 8 connection(s)
I20260806 11:45:35.656256 25289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:35.659022 25387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:35.664623 25387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468: Bootstrap starting.
I20260806 11:45:35.666357 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 25289
I20260806 11:45:35.666508 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:35.666855 452 external_mini_cluster.cc:1428] Setting key d0ebbe9ef19e03da862d40e44c0870f2
I20260806 11:45:35.667471 25387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:35.668406 25387 log.cc:824] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:35.670295 25387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468: No bootstrap required, opened a new log
I20260806 11:45:35.673873 25387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf6319b1c814a1f8d3b6410e8c9c468" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37551 } }
I20260806 11:45:35.674199 25387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:35.674288 25387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf6319b1c814a1f8d3b6410e8c9c468, State: Initialized, Role: FOLLOWER
I20260806 11:45:35.674727 25387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf6319b1c814a1f8d3b6410e8c9c468" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37551 } }
I20260806 11:45:35.674896 25387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:35.674989 25387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:35.675091 25387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 11:45:35 dist-test-slave-45dx krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:45:35.676071 25387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf6319b1c814a1f8d3b6410e8c9c468" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37551 } }
I20260806 11:45:35.676469 25387 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf6319b1c814a1f8d3b6410e8c9c468; no voters:
I20260806 11:45:35.676843 25387 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:35.676919 25393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:35.677150 25393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [term 1 LEADER]: Becoming Leader. State: Replica: 6bf6319b1c814a1f8d3b6410e8c9c468, State: Running, Role: LEADER
I20260806 11:45:35.677513 25393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf6319b1c814a1f8d3b6410e8c9c468" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37551 } }
I20260806 11:45:35.678181 25387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:35.678768 25393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf6319b1c814a1f8d3b6410e8c9c468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf6319b1c814a1f8d3b6410e8c9c468" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37551 } } }
I20260806 11:45:35.678985 25393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:35.679464 25409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:35.680400 25394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf6319b1c814a1f8d3b6410e8c9c468. Latest consensus state: current_term: 1 leader_uuid: "6bf6319b1c814a1f8d3b6410e8c9c468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf6319b1c814a1f8d3b6410e8c9c468" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37551 } } }
I20260806 11:45:35.680590 25394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:35.682552 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:35.668197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.0.113.62:37551)
0806 11:45:35.668719 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:35.668736 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:35.668770 (+ 34us) server_negotiation.cc:407] Connection header received
0806 11:45:35.669411 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:35.669428 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:35.669740 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:35.670097 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:35.670919 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:35.671899 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:35.672829 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:35.673298 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:35.673478 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:35.676266 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:35.676271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:35.676283 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:35.676374 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:35.678431 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:35.679036 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:35.679045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:35.679054 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:35.679220 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:35.679602 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:35.679606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:35.679611 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:35.680203 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:35.681088 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:35.681245 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 11:45:35.681558 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":150,"threads_started":1}
I20260806 11:45:35.684003 25409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:35.689831 25409 catalog_manager.cc:1347] Generated new cluster ID: 3a54277ef1964f1bbd075148d4906208
I20260806 11:45:35.689918 25409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:35.697518 25409 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:35.698510 25409 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:35.707655 25409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468: Generated new TSK 0
I20260806 11:45:35.708276 25409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:45:35.743453 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37551
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:35.885274 25424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:35.885538 25424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:35.885583 25424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:35.889514 25424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:35.889569 25424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:35.889624 25424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:45:35.894698 25424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:35.895737 25424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:35.896750 25424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:35.906077 25430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:35.906111 25433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:35.906116 25431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:35.906278 25424 server_base.cc:1034] running on GCE node
I20260806 11:45:35.906812 25424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:35.907436 25424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:35.908618 25424 hybrid_clock.cc:648] HybridClock initialized: now 1786016735908606 us; error 24 us; skew 500 ppm
Aug 06 11:45:35 dist-test-slave-45dx krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:35.911386 25424 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:45:35.912134 25424 webserver.cc:460] Webserver started at http://127.0.113.1:35021/ using document root <none> and password file <none>
I20260806 11:45:35.912719 25424 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:35.912781 25424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:35.912937 25424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:35.914837 25424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "22abb0b679e841afb1bf0ab0faaaa45a"
format_stamp: "Formatted at 2026-08-06 11:45:35 on dist-test-slave-45dx"
server_key: "7f181b0fee410996bb4c6d0371b971c0"
server_key_iv: "30d478896f19897f853a05907957c177"
server_key_version: "encryptionkey@0"
I20260806 11:45:35.915325 25424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "22abb0b679e841afb1bf0ab0faaaa45a"
format_stamp: "Formatted at 2026-08-06 11:45:35 on dist-test-slave-45dx"
server_key: "7f181b0fee410996bb4c6d0371b971c0"
server_key_iv: "30d478896f19897f853a05907957c177"
server_key_version: "encryptionkey@0"
I20260806 11:45:35.918865 25424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 11:45:35.921211 25443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:35.922394 25424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260806 11:45:35.922487 25424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "22abb0b679e841afb1bf0ab0faaaa45a"
format_stamp: "Formatted at 2026-08-06 11:45:35 on dist-test-slave-45dx"
server_key: "7f181b0fee410996bb4c6d0371b971c0"
server_key_iv: "30d478896f19897f853a05907957c177"
server_key_version: "encryptionkey@0"
I20260806 11:45:35.922569 25424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:35.934106 25424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:35.937727 25424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:35.937881 25424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:35.938413 25424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:35.939169 25424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:35.939215 25424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:35.939254 25424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:35.939275 25424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:35.947898 25424 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:35347
I20260806 11:45:35.947928 25606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:35347 every 8 connection(s)
I20260806 11:45:35.948827 25424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:45:35.951375 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 25424
I20260806 11:45:35.951539 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:45:35.951893 452 external_mini_cluster.cc:1428] Setting key 55323125c46b23bc916647295b935bea
Aug 06 11:45:35 dist-test-slave-45dx krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:45:35.963665 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:35.950565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:53459 (local address 127.0.113.62:37551)
0806 11:45:35.950707 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:35.950710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:35.951582 (+ 872us) server_negotiation.cc:407] Connection header received
0806 11:45:35.952808 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:35.952812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:35.952859 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:35.952967 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:35.954508 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:35.955062 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:35.956491 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:35.956916 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:35.957056 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:35.959941 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:35.959944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:35.959946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:35.959970 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:35.961455 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:35.961986 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:35.961990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:35.961992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:35.962079 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:35.962437 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:35.962442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:35.962444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:35.962625 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:35.962734 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:35.963299 (+ 565us) server_negotiation.cc:299] Negotiation successful
0806 11:45:35.963437 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:45:35.964561 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:35.950932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37551 (local address 127.0.113.1:53459)
0806 11:45:35.951409 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:35.951470 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:35.952562 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:35.953150 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:35.953156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:35.953668 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:35.954357 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:35.954378 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:35.955191 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:35.955197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:35.956365 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:35.956385 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:35.957080 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:35.957083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:35.957221 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:35.957843 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:35.957860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:35.959767 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:35.961582 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:35.961588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:35.961601 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:35.961880 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:35.962171 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:35.962174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:35.962177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:35.962322 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:35.962748 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:35.962756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:35.963071 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 11:45:35.963257 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 11:45:35.963600 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:45:35.965777 25608 heartbeater.cc:347] Connected to a master server at 127.0.113.62:37551
I20260806 11:45:35.966027 25608 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:35.966536 25608 heartbeater.cc:511] Master 127.0.113.62:37551 requested a full tablet report, sending...
I20260806 11:45:35.968259 25338 ts_manager.cc:194] Registered new tserver with Master: 22abb0b679e841afb1bf0ab0faaaa45a (127.0.113.1:35347)
I20260806 11:45:35.969563 25338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:53459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:45:35.983148 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37551
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:36.094594 25616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:36.094911 25616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:36.094955 25616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:36.098963 25616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:36.099025 25616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:36.099086 25616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:45:36.104435 25616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:36.105384 25616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:36.106336 25616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:36.115900 25621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:36.115863 25624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:36.115864 25622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:36.116024 25616 server_base.cc:1034] running on GCE node
I20260806 11:45:36.116707 25616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:36.117370 25616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:36.118568 25616 hybrid_clock.cc:648] HybridClock initialized: now 1786016736118531 us; error 47 us; skew 500 ppm
Aug 06 11:45:36 dist-test-slave-45dx krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:36.121549 25616 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:45:36.122443 25616 webserver.cc:460] Webserver started at http://127.0.113.2:45663/ using document root <none> and password file <none>
I20260806 11:45:36.123015 25616 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:36.123107 25616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:36.123356 25616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:36.125505 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "f6111bc2d5304633af248d6efae21a7b"
format_stamp: "Formatted at 2026-08-06 11:45:36 on dist-test-slave-45dx"
server_key: "d33767df274f8635c17487cb60055293"
server_key_iv: "5ee0125123a7b6e02e198d26451555f4"
server_key_version: "encryptionkey@0"
I20260806 11:45:36.126143 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "f6111bc2d5304633af248d6efae21a7b"
format_stamp: "Formatted at 2026-08-06 11:45:36 on dist-test-slave-45dx"
server_key: "d33767df274f8635c17487cb60055293"
server_key_iv: "5ee0125123a7b6e02e198d26451555f4"
server_key_version: "encryptionkey@0"
I20260806 11:45:36.129982 25616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 11:45:36.132596 25634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:36.133922 25616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 11:45:36.134107 25616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "f6111bc2d5304633af248d6efae21a7b"
format_stamp: "Formatted at 2026-08-06 11:45:36 on dist-test-slave-45dx"
server_key: "d33767df274f8635c17487cb60055293"
server_key_iv: "5ee0125123a7b6e02e198d26451555f4"
server_key_version: "encryptionkey@0"
I20260806 11:45:36.134238 25616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:36.156530 25616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:36.159469 25616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:36.159684 25616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:36.160290 25616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:36.161197 25616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:36.161270 25616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:36.161339 25616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:36.161381 25616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:36.170428 25616 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:40103
I20260806 11:45:36.170456 25784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:40103 every 8 connection(s)
I20260806 11:45:36.171311 25616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:45:36.180068 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 25616
I20260806 11:45:36.180197 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:45:36.180622 452 external_mini_cluster.cc:1428] Setting key f91d4df50d65ac1feb5eade14a2f78b9
Aug 06 11:45:36 dist-test-slave-45dx krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:45:36.186177 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.173235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:53657 (local address 127.0.113.62:37551)
0806 11:45:36.173415 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.173420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.174206 (+ 786us) server_negotiation.cc:407] Connection header received
0806 11:45:36.175480 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.175483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.175534 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.175613 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:36.176843 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.177394 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.178700 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.178960 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.179047 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.182163 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:36.182167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:36.182170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:36.182201 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:36.183773 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:36.184530 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:36.184535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:36.184541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:36.184630 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:36.185014 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:36.185017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:36.185019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:36.185211 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:36.185319 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.185862 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.186001 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 11:45:36.187072 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.173574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37551 (local address 127.0.113.2:53657)
0806 11:45:36.174051 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:36.174094 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:36.175252 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:36.175779 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:36.175785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:36.176166 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:36.176709 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.176723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.177518 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.177521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.178445 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.178455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.179117 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.179120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.179267 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.179988 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:36.180005 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:36.181988 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:36.183911 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:36.183921 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:36.183940 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:36.184381 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:36.184739 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:36.184742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:36.184744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:36.184899 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:36.185340 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:36.185347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:36.185602 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 11:45:36.185788 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 11:45:36.186131 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":107,"threads_started":1}
I20260806 11:45:36.188447 25786 heartbeater.cc:347] Connected to a master server at 127.0.113.62:37551
I20260806 11:45:36.188709 25786 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:36.189222 25786 heartbeater.cc:511] Master 127.0.113.62:37551 requested a full tablet report, sending...
I20260806 11:45:36.190394 25338 ts_manager.cc:194] Registered new tserver with Master: f6111bc2d5304633af248d6efae21a7b (127.0.113.2:40103)
I20260806 11:45:36.191241 25338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:53657
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:45:36.211812 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37551
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:36.329116 25791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:36.329399 25791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:36.329443 25791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:36.333493 25791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:36.333556 25791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:36.333614 25791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:45:36.339519 25791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:36.340685 25791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:36.341701 25791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:36.351460 25799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:36.351480 25798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:36.351544 25791 server_base.cc:1034] running on GCE node
W20260806 11:45:36.351487 25802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:36.352161 25791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:36.352869 25791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:36.354054 25791 hybrid_clock.cc:648] HybridClock initialized: now 1786016736353975 us; error 90 us; skew 500 ppm
Aug 06 11:45:36 dist-test-slave-45dx krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:36.356985 25791 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:45:36.357770 25791 webserver.cc:460] Webserver started at http://127.0.113.3:34805/ using document root <none> and password file <none>
I20260806 11:45:36.358280 25791 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:36.358340 25791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:36.358502 25791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:36.360656 25791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "eb8323787a81427a8e18e5149a48e9c6"
format_stamp: "Formatted at 2026-08-06 11:45:36 on dist-test-slave-45dx"
server_key: "d7e4ca88df0b01d95345ab8718de2726"
server_key_iv: "aed2015214f358f3e8aa6f6478e9da18"
server_key_version: "encryptionkey@0"
I20260806 11:45:36.361225 25791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "eb8323787a81427a8e18e5149a48e9c6"
format_stamp: "Formatted at 2026-08-06 11:45:36 on dist-test-slave-45dx"
server_key: "d7e4ca88df0b01d95345ab8718de2726"
server_key_iv: "aed2015214f358f3e8aa6f6478e9da18"
server_key_version: "encryptionkey@0"
I20260806 11:45:36.365403 25791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 11:45:36.368158 25812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:36.369679 25791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 11:45:36.369835 25791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "eb8323787a81427a8e18e5149a48e9c6"
format_stamp: "Formatted at 2026-08-06 11:45:36 on dist-test-slave-45dx"
server_key: "d7e4ca88df0b01d95345ab8718de2726"
server_key_iv: "aed2015214f358f3e8aa6f6478e9da18"
server_key_version: "encryptionkey@0"
I20260806 11:45:36.369973 25791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:36.384460 25791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:36.388262 25791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:36.388562 25791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:36.389319 25791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:36.390193 25791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:36.390270 25791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:36.390352 25791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:36.390405 25791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:36.400028 25791 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:39849
I20260806 11:45:36.400050 25980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:39849 every 8 connection(s)
I20260806 11:45:36.401050 25791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:45:36.409773 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 25791
I20260806 11:45:36.409937 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:45:36.410279 452 external_mini_cluster.cc:1428] Setting key fdcee0a2f5212bf3796f81ad32f40d0c
Aug 06 11:45:36 dist-test-slave-45dx krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:45:36.415511 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.403186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:33561 (local address 127.0.113.62:37551)
0806 11:45:36.403327 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.403331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.404190 (+ 859us) server_negotiation.cc:407] Connection header received
0806 11:45:36.405550 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.405554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.405617 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.405721 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:36.406922 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.407463 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.408663 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.408890 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.409004 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.411926 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:36.411951 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:36.411956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:36.411988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:36.413616 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:36.414127 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:36.414130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:36.414131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:36.414212 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:36.414560 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:36.414562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:36.414564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:36.414737 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:36.414813 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.415295 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.415395 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:45:36.416549 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.403512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37551 (local address 127.0.113.3:33561)
0806 11:45:36.404013 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:36.404064 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:36.405320 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:36.405900 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:36.405906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:36.406283 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:36.406760 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.406774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.407598 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.407602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.408514 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.408524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.409047 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.409052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.409228 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.409818 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:36.409853 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:36.411685 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:36.413740 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:36.413745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:36.413756 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:36.414029 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:36.414316 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:36.414319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:36.414321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:36.414472 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:36.414828 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:36.414834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:36.415089 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 11:45:36.415245 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 11:45:36.415502 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260806 11:45:36.417801 25981 heartbeater.cc:347] Connected to a master server at 127.0.113.62:37551
I20260806 11:45:36.418067 25981 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:36.418581 25981 heartbeater.cc:511] Master 127.0.113.62:37551 requested a full tablet report, sending...
I20260806 11:45:36.419710 25338 ts_manager.cc:194] Registered new tserver with Master: eb8323787a81427a8e18e5149a48e9c6 (127.0.113.3:39849)
I20260806 11:45:36.420413 25338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:33561
I20260806 11:45:36.424590 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:45:36.433775 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.426589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.0.113.62:37551)
0806 11:45:36.426713 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.426717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.426935 (+ 218us) server_negotiation.cc:407] Connection header received
0806 11:45:36.427068 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.427073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.427118 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.427185 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:36.427825 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.428364 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.429180 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.429391 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.429466 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.430774 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:36.430777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:36.430779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:36.430801 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:36.432218 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:36.432681 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:36.432684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:36.432685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:36.432762 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:36.432996 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:36.432998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:36.432999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:36.433192 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:36.433296 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.433521 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.433613 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:45:36.436852 25338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:45:36.440140 25338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:45:36.456607 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.449596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:40103 (local address 127.0.0.1:42980)
0806 11:45:36.450812 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:36.450827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:36.450919 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:36.451569 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:36.451572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:36.451590 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:36.451786 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.451793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.453207 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.453211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.454947 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.454959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.455947 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.455954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.456088 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.456132 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 11:45:36.456263 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 11:45:36.456471 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"mutex_wait_us":193,"spinlock_wait_cycles":768,"thread_start_us":126,"threads_started":1}
I20260806 11:45:36.456612 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.450524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.113.1:35347)
0806 11:45:36.452004 (+ 1480us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.452040 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.452067 (+ 27us) server_negotiation.cc:407] Connection header received
0806 11:45:36.452145 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.452151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.452363 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.452544 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:36.452799 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.454116 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.455265 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.455887 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.456003 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.456160 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.456286 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.456429 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":70,"threads_started":1}
I20260806 11:45:36.457211 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.450319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.0.113.2:40103)
0806 11:45:36.451194 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.451204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.451224 (+ 20us) server_negotiation.cc:407] Connection header received
0806 11:45:36.451285 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.451288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.451423 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.451594 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:36.451914 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.453011 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.455107 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.455789 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.455887 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.456055 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.456920 (+ 865us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.457028 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":106,"threads_started":1}
I20260806 11:45:36.457479 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.449906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:35347 (local address 127.0.0.1:51850)
0806 11:45:36.450910 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:36.450925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:36.451020 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:36.452542 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:36.452546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:36.452573 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:36.452663 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.452669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.454256 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.454262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.455128 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.455139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.456026 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.456030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.456121 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.456145 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 11:45:36.456256 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 11:45:36.456447 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":58,"thread_start_us":101,"threads_started":1}
I20260806 11:45:36.457739 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.449568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:39849 (local address 127.0.0.1:45796)
0806 11:45:36.450464 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:36.450520 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:36.450651 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:36.451553 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:36.451560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:36.451582 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:36.451783 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.451793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.453393 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.453396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.454947 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.454959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.457313 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.457319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.457407 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.457443 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 11:45:36.457518 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 11:45:36.457597 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":144,"spinlock_wait_cycles":8064,"thread_start_us":121,"threads_started":1}
I20260806 11:45:36.458245 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.449761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.0.113.3:39849)
0806 11:45:36.451017 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.451030 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.451055 (+ 25us) server_negotiation.cc:407] Connection header received
0806 11:45:36.451141 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.451146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.451398 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.451566 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:36.452041 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.453228 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.456127 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.456981 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.457088 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.457246 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.458004 (+ 758us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.458097 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1134,"thread_start_us":168,"threads_started":1}
I20260806 11:45:36.460527 25701 tablet_service.cc:1467] Processing CreateTablet for tablet 03de8bf51899439ab5d329364189e2cd (DEFAULT_TABLE table=test-table [id=b317387818ec4deabd8bef4310dc72db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:36.460527 25513 tablet_service.cc:1467] Processing CreateTablet for tablet 03de8bf51899439ab5d329364189e2cd (DEFAULT_TABLE table=test-table [id=b317387818ec4deabd8bef4310dc72db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:36.460719 25885 tablet_service.cc:1467] Processing CreateTablet for tablet 03de8bf51899439ab5d329364189e2cd (DEFAULT_TABLE table=test-table [id=b317387818ec4deabd8bef4310dc72db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:36.461704 25885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03de8bf51899439ab5d329364189e2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:36.461711 25701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03de8bf51899439ab5d329364189e2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:36.461711 25513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03de8bf51899439ab5d329364189e2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:36.467962 26004 tablet_bootstrap.cc:492] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: Bootstrap starting.
I20260806 11:45:36.469609 26005 tablet_bootstrap.cc:492] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a: Bootstrap starting.
I20260806 11:45:36.470676 26004 tablet_bootstrap.cc:654] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:36.471773 26005 tablet_bootstrap.cc:654] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:36.472210 26004 log.cc:824] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:36.472636 26005 log.cc:824] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:36.472678 26007 tablet_bootstrap.cc:492] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: Bootstrap starting.
I20260806 11:45:36.474419 26004 tablet_bootstrap.cc:492] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: No bootstrap required, opened a new log
I20260806 11:45:36.474634 26004 ts_tablet_manager.cc:1397] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260806 11:45:36.474979 26007 tablet_bootstrap.cc:654] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:36.475643 26007 log.cc:824] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:36.477595 26007 tablet_bootstrap.cc:492] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: No bootstrap required, opened a new log
I20260806 11:45:36.477851 26007 ts_tablet_manager.cc:1397] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:45:36.478035 26005 tablet_bootstrap.cc:492] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a: No bootstrap required, opened a new log
I20260806 11:45:36.478233 26005 ts_tablet_manager.cc:1397] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260806 11:45:36.478540 26004 raft_consensus.cc:348] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } }
I20260806 11:45:36.478838 26004 raft_consensus.cc:374] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:36.478889 26004 raft_consensus.cc:729] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8323787a81427a8e18e5149a48e9c6, State: Initialized, Role: FOLLOWER
I20260806 11:45:36.479439 26004 consensus_queue.cc:260] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } }
I20260806 11:45:36.480168 25981 heartbeater.cc:503] Master 127.0.113.62:37551 was elected leader, sending a full tablet report...
I20260806 11:45:36.480679 26004 ts_tablet_manager.cc:1428] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260806 11:45:36.481789 26007 raft_consensus.cc:348] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } }
I20260806 11:45:36.482079 26007 raft_consensus.cc:374] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:36.482168 26007 raft_consensus.cc:729] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6111bc2d5304633af248d6efae21a7b, State: Initialized, Role: FOLLOWER
I20260806 11:45:36.482640 26007 consensus_queue.cc:260] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } }
I20260806 11:45:36.483345 25786 heartbeater.cc:503] Master 127.0.113.62:37551 was elected leader, sending a full tablet report...
I20260806 11:45:36.483690 26007 ts_tablet_manager.cc:1428] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 11:45:36.483042 26005 raft_consensus.cc:348] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } }
I20260806 11:45:36.484503 26005 raft_consensus.cc:374] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:36.484571 26005 raft_consensus.cc:729] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22abb0b679e841afb1bf0ab0faaaa45a, State: Initialized, Role: FOLLOWER
I20260806 11:45:36.485045 26005 consensus_queue.cc:260] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } }
I20260806 11:45:36.486125 26005 ts_tablet_manager.cc:1428] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260806 11:45:36.486534 25608 heartbeater.cc:503] Master 127.0.113.62:37551 was elected leader, sending a full tablet report...
I20260806 11:45:36.510000 26017 raft_consensus.cc:482] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:36.510226 26017 raft_consensus.cc:504] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } }
I20260806 11:45:36.511425 26017 leader_election.cc:288] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb8323787a81427a8e18e5149a48e9c6 (127.0.113.3:39849), 22abb0b679e841afb1bf0ab0faaaa45a (127.0.113.1:35347)
I20260806 11:45:36.515192 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.511686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:35347 (local address 127.0.113.2:51779)
0806 11:45:36.511806 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:36.511822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:36.511918 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:36.512359 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:36.512363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:36.512382 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:36.512517 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.512528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.513387 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.513392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.514146 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.514154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.514780 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.514783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.514893 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.514911 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 11:45:36.514996 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 11:45:36.515067 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 11:45:36.515192 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.511845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:51779 (local address 127.0.113.1:35347)
0806 11:45:36.512015 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.512032 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.512050 (+ 18us) server_negotiation.cc:407] Connection header received
0806 11:45:36.512129 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.512135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.512224 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.512348 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:36.512650 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.513261 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.514271 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.514659 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.514776 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.514834 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.515013 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.515076 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 11:45:36.515548 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.511686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:39849 (local address 127.0.113.2:60045)
0806 11:45:36.512457 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:36.512473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:36.512574 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:36.512858 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:36.512861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:36.512876 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:36.512990 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.512995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.513694 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.513699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.514445 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:36.514453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.515080 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.515084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:36.515184 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.515202 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 11:45:36.515297 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 11:45:36.515384 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":278,"thread_start_us":217,"threads_started":1}
I20260806 11:45:36.515640 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.511887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:60045 (local address 127.0.113.3:39849)
0806 11:45:36.512064 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.512068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.512566 (+ 498us) server_negotiation.cc:407] Connection header received
0806 11:45:36.512686 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.512695 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.512737 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.512819 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:36.513107 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.513588 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.514574 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.514961 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.515076 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.515132 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.515426 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.515499 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:45:36.515880 25539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03de8bf51899439ab5d329364189e2cd" candidate_uuid: "f6111bc2d5304633af248d6efae21a7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" is_pre_election: true
I20260806 11:45:36.516214 25539 raft_consensus.cc:2454] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6111bc2d5304633af248d6efae21a7b in term 0.
I20260806 11:45:36.516384 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03de8bf51899439ab5d329364189e2cd" candidate_uuid: "f6111bc2d5304633af248d6efae21a7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb8323787a81427a8e18e5149a48e9c6" is_pre_election: true
I20260806 11:45:36.516682 25919 raft_consensus.cc:2454] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6111bc2d5304633af248d6efae21a7b in term 0.
I20260806 11:45:36.516763 25637 leader_election.cc:302] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22abb0b679e841afb1bf0ab0faaaa45a, f6111bc2d5304633af248d6efae21a7b; no voters:
I20260806 11:45:36.517104 26017 raft_consensus.cc:2781] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:45:36.517202 26017 raft_consensus.cc:482] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:45:36.517247 26017 raft_consensus.cc:3037] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:36.518286 26017 raft_consensus.cc:504] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } }
I20260806 11:45:36.518657 26017 leader_election.cc:288] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [CANDIDATE]: Term 1 election: Requested vote from peers eb8323787a81427a8e18e5149a48e9c6 (127.0.113.3:39849), 22abb0b679e841afb1bf0ab0faaaa45a (127.0.113.1:35347)
I20260806 11:45:36.519009 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03de8bf51899439ab5d329364189e2cd" candidate_uuid: "f6111bc2d5304633af248d6efae21a7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb8323787a81427a8e18e5149a48e9c6"
I20260806 11:45:36.519076 25539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03de8bf51899439ab5d329364189e2cd" candidate_uuid: "f6111bc2d5304633af248d6efae21a7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22abb0b679e841afb1bf0ab0faaaa45a"
I20260806 11:45:36.519140 25919 raft_consensus.cc:3037] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:36.519184 25539 raft_consensus.cc:3037] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:36.520489 25539 raft_consensus.cc:2454] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6111bc2d5304633af248d6efae21a7b in term 1.
I20260806 11:45:36.520489 25919 raft_consensus.cc:2454] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6111bc2d5304633af248d6efae21a7b in term 1.
I20260806 11:45:36.520812 25638 leader_election.cc:302] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb8323787a81427a8e18e5149a48e9c6, f6111bc2d5304633af248d6efae21a7b; no voters:
I20260806 11:45:36.521018 26017 raft_consensus.cc:2781] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:36.521253 26017 raft_consensus.cc:686] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 1 LEADER]: Becoming Leader. State: Replica: f6111bc2d5304633af248d6efae21a7b, State: Running, Role: LEADER
I20260806 11:45:36.521593 26017 consensus_queue.cc:237] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } }
I20260806 11:45:36.525147 25337 catalog_manager.cc:5495] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b reported cstate change: term changed from 0 to 1, leader changed from <none> to f6111bc2d5304633af248d6efae21a7b (127.0.113.2). New cstate: current_term: 1 leader_uuid: "f6111bc2d5304633af248d6efae21a7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6111bc2d5304633af248d6efae21a7b" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 } health_report { overall_health: UNKNOWN } } }
I20260806 11:45:36.557325 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.553271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.0.113.2:40103)
0806 11:45:36.553433 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.553437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.553453 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:45:36.553584 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.553587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.553644 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.553741 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:45:36.554069 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.554572 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.555570 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.555806 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.555899 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.556252 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:45:36.556809 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:45:36.556915 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.557053 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.557114 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 11:45:36.564077 25539 raft_consensus.cc:1261] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a [term 1 FOLLOWER]: Refusing update from remote peer f6111bc2d5304633af248d6efae21a7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:36.564070 25919 raft_consensus.cc:1261] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [term 1 FOLLOWER]: Refusing update from remote peer f6111bc2d5304633af248d6efae21a7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:36.565038 26022 consensus_queue.cc:1035] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "22abb0b679e841afb1bf0ab0faaaa45a" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:36.565266 26017 consensus_queue.cc:1035] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb8323787a81427a8e18e5149a48e9c6" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:36.583228 25337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:36.583447 25337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:36.586117 25337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bf6319b1c814a1f8d3b6410e8c9c468: Sending DeleteTablet for 3 replicas of tablet 03de8bf51899439ab5d329364189e2cd
I20260806 11:45:36.587011 25701 tablet_service.cc:1514] Processing DeleteTablet for tablet 03de8bf51899439ab5d329364189e2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:36 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:42980
I20260806 11:45:36.587406 25513 tablet_service.cc:1514] Processing DeleteTablet for tablet 03de8bf51899439ab5d329364189e2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:36 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:51850
I20260806 11:45:36.587558 25885 tablet_service.cc:1514] Processing DeleteTablet for tablet 03de8bf51899439ab5d329364189e2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:36 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:45796
I20260806 11:45:36.587944 26037 tablet_replica.cc:330] T 03de8bf51899439ab5d329364189e2cd P 22abb0b679e841afb1bf0ab0faaaa45a: stopping tablet replica
I20260806 11:45:36.588121 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 25424
I20260806 11:45:36.594762 26036 tablet_replica.cc:330] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: stopping tablet replica
I20260806 11:45:36.595183 26036 raft_consensus.cc:2229] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:45:36.595625 26036 raft_consensus.cc:2258] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:36.598169 26036 ts_tablet_manager.cc:1905] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:36.601243 26036 ts_tablet_manager.cc:1918] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:36.601358 26036 log.cc:1196] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/03de8bf51899439ab5d329364189e2cd
I20260806 11:45:36.601697 26036 ts_tablet_manager.cc:1939] T 03de8bf51899439ab5d329364189e2cd P f6111bc2d5304633af248d6efae21a7b: Deleting consensus metadata
I20260806 11:45:36.603070 25319 catalog_manager.cc:4841] TS f6111bc2d5304633af248d6efae21a7b (127.0.113.2:40103): tablet 03de8bf51899439ab5d329364189e2cd (table test-table [id=b317387818ec4deabd8bef4310dc72db]) successfully deleted
I20260806 11:45:36.604673 26038 tablet_replica.cc:330] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: stopping tablet replica
I20260806 11:45:36.605044 26038 raft_consensus.cc:2229] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:45:36.605394 26038 raft_consensus.cc:2258] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:36.607096 26038 ts_tablet_manager.cc:1905] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 11:45:36.607851 25323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:35347 (error 108)
I20260806 11:45:36.608981 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 25616
I20260806 11:45:36.609045 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.608694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:35347 (local address 127.0.0.1:51866)
0806 11:45:36.608851 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:36.608941 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 11:45:36.609794 26038 ts_tablet_manager.cc:1918] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:36.609918 26038 log.cc:1196] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/03de8bf51899439ab5d329364189e2cd
I20260806 11:45:36.610214 26038 ts_tablet_manager.cc:1939] T 03de8bf51899439ab5d329364189e2cd P eb8323787a81427a8e18e5149a48e9c6: Deleting consensus metadata
I20260806 11:45:36.611385 25324 catalog_manager.cc:4841] TS eb8323787a81427a8e18e5149a48e9c6 (127.0.113.3:39849): tablet 03de8bf51899439ab5d329364189e2cd (table test-table [id=b317387818ec4deabd8bef4310dc72db]) successfully deleted
W20260806 11:45:36.619722 25323 catalog_manager.cc:4584] TS 22abb0b679e841afb1bf0ab0faaaa45a (127.0.113.1:35347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03de8bf51899439ab5d329364189e2cd: Network error: Client connection negotiation failed: client connection to 127.0.113.1:35347: connect: Connection refused (error 111)
I20260806 11:45:36.625928 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 25791
I20260806 11:45:36.636759 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 25289
2026-08-06T11:45:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2543 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T11:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:36Z Disabled control of system clock
I20260806 11:45:36.675112 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:37057
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:37057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 11:45:36.787390 26044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:36.787686 26044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:36.787724 26044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:36.791782 26044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:36.791846 26044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:36.791862 26044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:36.791880 26044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:36.791934 26044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:36.796649 26044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:37057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:37057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:36.797891 26044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:36.799435 26044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:36.808780 26052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:36.808784 26056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:36.808784 26053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:36.809130 26044 server_base.cc:1034] running on GCE node
I20260806 11:45:36.809803 26044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:36.811033 26044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:36.812216 26044 hybrid_clock.cc:648] HybridClock initialized: now 1786016736812198 us; error 31 us; skew 500 ppm
I20260806 11:45:36.814155 26044 webserver.cc:460] Webserver started at http://127.0.113.62:46491/ using document root <none> and password file <none>
I20260806 11:45:36.814697 26044 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:36.814756 26044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:36.814952 26044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:36.817096 26044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "bc04ec8685be4f1183154a01a42918d3"
format_stamp: "Formatted at 2026-08-06 11:45:36 on dist-test-slave-45dx"
server_key: "1f8830078611c44b3aa87aba716722f6"
server_key_iv: "89906c7d632ccf55e249745a8ad71246"
server_key_version: "encryptionkey@0"
I20260806 11:45:36.817623 26044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "bc04ec8685be4f1183154a01a42918d3"
format_stamp: "Formatted at 2026-08-06 11:45:36 on dist-test-slave-45dx"
server_key: "1f8830078611c44b3aa87aba716722f6"
server_key_iv: "89906c7d632ccf55e249745a8ad71246"
server_key_version: "encryptionkey@0"
I20260806 11:45:36.821345 26044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:45:36.823812 26064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:36.825181 26044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:45:36.825287 26044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "bc04ec8685be4f1183154a01a42918d3"
format_stamp: "Formatted at 2026-08-06 11:45:36 on dist-test-slave-45dx"
server_key: "1f8830078611c44b3aa87aba716722f6"
server_key_iv: "89906c7d632ccf55e249745a8ad71246"
server_key_version: "encryptionkey@0"
I20260806 11:45:36.825376 26044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:36.858834 26044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:36.859565 26044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:36.859728 26044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:36.867126 26044 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:37057
I20260806 11:45:36.867138 26138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:37057 every 8 connection(s)
I20260806 11:45:36.868119 26044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:36.871433 26139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:36.871941 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 26044
I20260806 11:45:36.872045 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:36.872428 452 external_mini_cluster.cc:1428] Setting key 35a21a2dac3bee61108250905b4d08dc
I20260806 11:45:36.878018 26139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3: Bootstrap starting.
I20260806 11:45:36.880779 26139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:36.881711 26139 log.cc:824] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:36.882289 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:36.873991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.0.113.62:37057)
0806 11:45:36.874636 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:36.874651 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:36.874689 (+ 38us) server_negotiation.cc:407] Connection header received
0806 11:45:36.875350 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:36.875367 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:36.875767 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:36.876055 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:36.876483 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.877470 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.878655 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:36.879121 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:36.879240 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:36.880167 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:36.880173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:36.880186 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:36.880676 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:36.880731 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:36.880752 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:36.880829 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:36.881042 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 11:45:36.881384 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
I20260806 11:45:36.884004 26139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3: No bootstrap required, opened a new log
I20260806 11:45:36.887668 26139 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc04ec8685be4f1183154a01a42918d3" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37057 } }
I20260806 11:45:36.887908 26139 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:36.887943 26139 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc04ec8685be4f1183154a01a42918d3, State: Initialized, Role: FOLLOWER
I20260806 11:45:36.888358 26139 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc04ec8685be4f1183154a01a42918d3" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37057 } }
I20260806 11:45:36.888497 26139 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:36.888540 26139 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:36.888612 26139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:36.889515 26139 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc04ec8685be4f1183154a01a42918d3" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37057 } }
I20260806 11:45:36.889808 26139 leader_election.cc:302] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc04ec8685be4f1183154a01a42918d3; no voters:
I20260806 11:45:36.890081 26139 leader_election.cc:288] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:36.890218 26145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:36.890509 26145 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [term 1 LEADER]: Becoming Leader. State: Replica: bc04ec8685be4f1183154a01a42918d3, State: Running, Role: LEADER
I20260806 11:45:36.890878 26145 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc04ec8685be4f1183154a01a42918d3" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37057 } }
I20260806 11:45:36.891413 26139 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:36.892659 26147 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc04ec8685be4f1183154a01a42918d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc04ec8685be4f1183154a01a42918d3" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37057 } } }
I20260806 11:45:36.892797 26147 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:36.893173 26157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:36.893123 26149 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc04ec8685be4f1183154a01a42918d3. Latest consensus state: current_term: 1 leader_uuid: "bc04ec8685be4f1183154a01a42918d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc04ec8685be4f1183154a01a42918d3" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 37057 } } }
I20260806 11:45:36.896389 26149 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:36.896483 26157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:36.902302 26157 catalog_manager.cc:1347] Generated new cluster ID: d2289d1d4be54ff294602ea9b11b5d9d
I20260806 11:45:36.902385 26157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:36.911036 26157 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:36.911878 26157 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:36.920421 26157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3: Generated new TSK 0
I20260806 11:45:36.921123 26157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:36.928961 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37057
--builtin_ntp_servers=127.0.113.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 11:45:37.044534 26176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:37.044792 26176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:37.044832 26176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:37.048955 26176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:37.049010 26176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:37.049085 26176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:45:37.053697 26176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:37.054755 26176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:37.056185 26176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:37.065716 26188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:37.065747 26184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:37.065809 26176 server_base.cc:1034] running on GCE node
W20260806 11:45:37.065716 26185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:37.066432 26176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:37.067000 26176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:37.068184 26176 hybrid_clock.cc:648] HybridClock initialized: now 1786016737068165 us; error 30 us; skew 500 ppm
I20260806 11:45:37.070075 26176 webserver.cc:460] Webserver started at http://127.0.113.1:45163/ using document root <none> and password file <none>
I20260806 11:45:37.070591 26176 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:37.070652 26176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:37.070813 26176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:37.072969 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "a53be8a5d58746eb8a89c5f1deb120ae"
format_stamp: "Formatted at 2026-08-06 11:45:37 on dist-test-slave-45dx"
server_key: "6514bfeed550af8a986b991a8048b473"
server_key_iv: "b7145054a01fb8881cf687e46f79fd21"
server_key_version: "encryptionkey@0"
I20260806 11:45:37.073511 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "a53be8a5d58746eb8a89c5f1deb120ae"
format_stamp: "Formatted at 2026-08-06 11:45:37 on dist-test-slave-45dx"
server_key: "6514bfeed550af8a986b991a8048b473"
server_key_iv: "b7145054a01fb8881cf687e46f79fd21"
server_key_version: "encryptionkey@0"
I20260806 11:45:37.077204 26176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:45:37.079607 26196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:37.080912 26176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:37.081012 26176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "a53be8a5d58746eb8a89c5f1deb120ae"
format_stamp: "Formatted at 2026-08-06 11:45:37 on dist-test-slave-45dx"
server_key: "6514bfeed550af8a986b991a8048b473"
server_key_iv: "b7145054a01fb8881cf687e46f79fd21"
server_key_version: "encryptionkey@0"
I20260806 11:45:37.081101 26176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:37.106814 26176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:37.107492 26176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:37.107648 26176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:37.108218 26176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:37.109180 26176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:37.109226 26176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:37.109267 26176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:37.109282 26176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:37.117847 26176 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:41095
I20260806 11:45:37.117868 26350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:41095 every 8 connection(s)
I20260806 11:45:37.118750 26176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:45:37.126659 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 26176
I20260806 11:45:37.126787 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:45:37.127178 452 external_mini_cluster.cc:1428] Setting key 4f3e95c4ff7a85a0b241b330aa629e59
I20260806 11:45:37.128351 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.120633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:50337 (local address 127.0.113.62:37057)
0806 11:45:37.120806 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.120811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.121948 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 11:45:37.123298 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.123306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.123359 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.123441 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:37.124402 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.124934 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.126063 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.126278 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.126367 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.127753 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:37.127758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:37.127761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:37.127776 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:37.127799 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:37.127810 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:37.127887 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.128098 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.128160 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 11:45:37.129317 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37057
--builtin_ntp_servers=127.0.113.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 11:45:37.129348 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.121091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37057 (local address 127.0.113.1:50337)
0806 11:45:37.121750 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:37.121806 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:37.123106 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:37.123603 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:37.123610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:37.123703 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:37.124227 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.124239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.125045 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.125048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.125949 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.125958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.126459 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.126463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.126674 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.127440 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:45:37.127442 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:37.127496 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:45:37.127575 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:37.127914 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:37.127925 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:37.127931 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 11:45:37.128066 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 11:45:37.128412 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260806 11:45:37.130708 26351 heartbeater.cc:347] Connected to a master server at 127.0.113.62:37057
I20260806 11:45:37.130952 26351 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:37.131614 26351 heartbeater.cc:511] Master 127.0.113.62:37057 requested a full tablet report, sending...
I20260806 11:45:37.133226 26091 ts_manager.cc:194] Registered new tserver with Master: a53be8a5d58746eb8a89c5f1deb120ae (127.0.113.1:41095)
I20260806 11:45:37.134389 26091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.1:50337
W20260806 11:45:37.244465 26360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:37.244750 26360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:37.244819 26360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:37.249991 26360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:37.250053 26360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:37.250140 26360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:45:37.254959 26360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:37.256026 26360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:37.257448 26360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:37.267100 26374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:37.267113 26368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:37.267098 26372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:37.267441 26360 server_base.cc:1034] running on GCE node
I20260806 11:45:37.267884 26360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:37.268574 26360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:37.269759 26360 hybrid_clock.cc:648] HybridClock initialized: now 1786016737269741 us; error 27 us; skew 500 ppm
I20260806 11:45:37.271597 26360 webserver.cc:460] Webserver started at http://127.0.113.2:43951/ using document root <none> and password file <none>
I20260806 11:45:37.272202 26360 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:37.272295 26360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:37.272534 26360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:37.274610 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "25ec6e8ec043436280a4bce1a741c568"
format_stamp: "Formatted at 2026-08-06 11:45:37 on dist-test-slave-45dx"
server_key: "fab5edd54541d01dd07c2fc20e9d05cc"
server_key_iv: "98856bba11199236e1a2603fc7250477"
server_key_version: "encryptionkey@0"
I20260806 11:45:37.275208 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "25ec6e8ec043436280a4bce1a741c568"
format_stamp: "Formatted at 2026-08-06 11:45:37 on dist-test-slave-45dx"
server_key: "fab5edd54541d01dd07c2fc20e9d05cc"
server_key_iv: "98856bba11199236e1a2603fc7250477"
server_key_version: "encryptionkey@0"
I20260806 11:45:37.278936 26360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 11:45:37.281337 26381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:37.282526 26360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:45:37.282670 26360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "25ec6e8ec043436280a4bce1a741c568"
format_stamp: "Formatted at 2026-08-06 11:45:37 on dist-test-slave-45dx"
server_key: "fab5edd54541d01dd07c2fc20e9d05cc"
server_key_iv: "98856bba11199236e1a2603fc7250477"
server_key_version: "encryptionkey@0"
I20260806 11:45:37.282781 26360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:37.299341 26360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:37.300011 26360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:37.300206 26360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:37.300843 26360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:37.301801 26360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:37.301880 26360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:37.301954 26360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:37.302011 26360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:37.311046 26360 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:42643
I20260806 11:45:37.311090 26549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:42643 every 8 connection(s)
I20260806 11:45:37.311954 26360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:45:37.317378 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 26360
I20260806 11:45:37.317507 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:45:37.317812 452 external_mini_cluster.cc:1428] Setting key d09fc7ff6f6bfa37fa5605e824b72fe6
I20260806 11:45:37.319981 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37057
--builtin_ntp_servers=127.0.113.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 11:45:37.321491 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.313808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:39055 (local address 127.0.113.62:37057)
0806 11:45:37.313974 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.313977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.314986 (+ 1009us) server_negotiation.cc:407] Connection header received
0806 11:45:37.316468 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.316471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.316533 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.316641 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:37.317612 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.318209 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.319431 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.319642 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.319721 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.321029 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:37.321032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:37.321034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:37.321045 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:37.321066 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:37.321074 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:37.321132 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.321318 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.321369 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:45:37.322540 26553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.314184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37057 (local address 127.0.113.2:39055)
0806 11:45:37.314828 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:37.314868 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:37.316178 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:37.316810 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:37.316817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:37.316897 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:37.317390 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.317402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.318372 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.318375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.319294 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.319305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.319808 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.319815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.320041 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.320728 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:45:37.320732 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:37.320789 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:45:37.320869 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:37.321159 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:37.321171 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:37.321179 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 11:45:37.321302 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 11:45:37.321602 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":135,"threads_started":1}
I20260806 11:45:37.323788 26550 heartbeater.cc:347] Connected to a master server at 127.0.113.62:37057
I20260806 11:45:37.324048 26550 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:37.324628 26550 heartbeater.cc:511] Master 127.0.113.62:37057 requested a full tablet report, sending...
I20260806 11:45:37.325677 26091 ts_manager.cc:194] Registered new tserver with Master: 25ec6e8ec043436280a4bce1a741c568 (127.0.113.2:42643)
I20260806 11:45:37.326241 26091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.2:39055
W20260806 11:45:37.435397 26557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:37.435679 26557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:37.435753 26557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:37.439752 26557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:37.439832 26557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:37.439916 26557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:45:37.444710 26557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:37057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:37.445740 26557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:37.447211 26557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:37.456867 26566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:37.456873 26568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:37.456919 26557 server_base.cc:1034] running on GCE node
W20260806 11:45:37.456866 26565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:37.457616 26557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:37.458245 26557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:37.459476 26557 hybrid_clock.cc:648] HybridClock initialized: now 1786016737459458 us; error 28 us; skew 500 ppm
I20260806 11:45:37.461370 26557 webserver.cc:460] Webserver started at http://127.0.113.3:43639/ using document root <none> and password file <none>
I20260806 11:45:37.461977 26557 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:37.462080 26557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:37.462323 26557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:37.464419 26557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "87979f5131134c518b2cfe2d337b5412"
format_stamp: "Formatted at 2026-08-06 11:45:37 on dist-test-slave-45dx"
server_key: "ce47ef0f54e796059db49efe76088541"
server_key_iv: "2cb9dfec03d7c749d4650958661ac502"
server_key_version: "encryptionkey@0"
I20260806 11:45:37.464985 26557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "87979f5131134c518b2cfe2d337b5412"
format_stamp: "Formatted at 2026-08-06 11:45:37 on dist-test-slave-45dx"
server_key: "ce47ef0f54e796059db49efe76088541"
server_key_iv: "2cb9dfec03d7c749d4650958661ac502"
server_key_version: "encryptionkey@0"
I20260806 11:45:37.468662 26557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 11:45:37.471062 26577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:37.472268 26557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:37.472453 26557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "87979f5131134c518b2cfe2d337b5412"
format_stamp: "Formatted at 2026-08-06 11:45:37 on dist-test-slave-45dx"
server_key: "ce47ef0f54e796059db49efe76088541"
server_key_iv: "2cb9dfec03d7c749d4650958661ac502"
server_key_version: "encryptionkey@0"
I20260806 11:45:37.472565 26557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:37.483371 26557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:37.484030 26557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:37.484247 26557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:37.484899 26557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:37.486011 26557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:37.486083 26557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:37.486150 26557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:37.486191 26557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:37.495505 26557 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:33293
I20260806 11:45:37.495571 26746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:33293 every 8 connection(s)
I20260806 11:45:37.496472 26557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:45:37.497895 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 26557
I20260806 11:45:37.498023 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:45:37.498402 452 external_mini_cluster.cc:1428] Setting key e46dc5257ecdbc2fb79eb4d45c22af6b
I20260806 11:45:37.505699 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.498375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:43297 (local address 127.0.113.62:37057)
0806 11:45:37.498558 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.498563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.499716 (+ 1153us) server_negotiation.cc:407] Connection header received
0806 11:45:37.501206 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.501210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.501268 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.501349 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:37.502239 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.502783 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.503928 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.504140 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.504247 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.505223 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:37.505226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:37.505229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:37.505240 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:37.505266 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:37.505274 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:37.505344 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.505509 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.505567 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 11:45:37.506501 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.498832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:37057 (local address 127.0.113.3:43297)
0806 11:45:37.499569 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:37.499621 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:37.501001 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:37.501524 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:37.501530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:37.501610 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:37.502096 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.502109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.502917 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.502920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.503781 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.503791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.504288 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.504292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.504466 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.504989 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:45:37.504990 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:37.505025 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:45:37.505082 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:37.505359 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:37.505369 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:37.505388 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 11:45:37.505505 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 11:45:37.505747 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":140,"threads_started":1}
I20260806 11:45:37.507535 26747 heartbeater.cc:347] Connected to a master server at 127.0.113.62:37057
I20260806 11:45:37.507764 26747 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:37.508247 26747 heartbeater.cc:511] Master 127.0.113.62:37057 requested a full tablet report, sending...
I20260806 11:45:37.509254 26091 ts_manager.cc:194] Registered new tserver with Master: 87979f5131134c518b2cfe2d337b5412 (127.0.113.3:33293)
I20260806 11:45:37.509850 26091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.3:43297
I20260806 11:45:37.512425 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:45:37.517686 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.514383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.0.113.62:37057)
0806 11:45:37.514499 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.514503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.514667 (+ 164us) server_negotiation.cc:407] Connection header received
0806 11:45:37.514745 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.514747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.514783 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.514848 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:37.515119 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.515580 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.516366 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.516570 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.516646 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.517365 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:37.517368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:37.517370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:37.517381 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:37.517399 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:37.517406 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:37.517464 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.517543 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.517587 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:45:37.520430 26091 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:45:37.523360 26091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:45:37.537631 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.531895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:42643 (local address 127.0.0.1:55976)
0806 11:45:37.532212 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:37.532261 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:37.532444 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:37.533006 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:37.533015 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:37.533041 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:37.533187 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.533194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.534577 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.534580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.536029 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.536039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.537072 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.537076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.537182 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.537219 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 11:45:37.537329 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 11:45:37.537456 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":158,"spinlock_wait_cycles":12800,"thread_start_us":98,"threads_started":1}
I20260806 11:45:37.537644 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.532059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.0.113.2:42643)
0806 11:45:37.532515 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.532528 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.532553 (+ 25us) server_negotiation.cc:407] Connection header received
0806 11:45:37.532629 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.532634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.532833 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.532998 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:37.533320 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.534450 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.536182 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.536937 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.537052 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.537230 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.537368 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.537466 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":110,"threads_started":1}
I20260806 11:45:37.538017 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.532226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:33293 (local address 127.0.0.1:33642)
0806 11:45:37.533278 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:37.533292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:37.533396 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:37.533895 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:37.533901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:37.533925 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:37.534037 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.534044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.535151 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.535155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.536180 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.536190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.537406 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.537410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.537487 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.537507 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 11:45:37.537574 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 11:45:37.537630 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":141,"threads_started":1}
I20260806 11:45:37.538285 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.532683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.0.113.3:33293)
0806 11:45:37.533007 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.533018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.533490 (+ 472us) server_negotiation.cc:407] Connection header received
0806 11:45:37.533564 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.533569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.533731 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.533870 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:37.534154 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.535018 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.536340 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.537060 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.537162 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.537346 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.537714 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.538025 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
I20260806 11:45:37.538385 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.531517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:41095 (local address 127.0.0.1:54748)
0806 11:45:37.532206 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:37.532255 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:37.532444 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:37.534008 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:37.534014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:37.534029 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:37.534112 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.534119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.535544 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.535547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.536665 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.536675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.537964 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.537968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.538071 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.538096 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 11:45:37.538182 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 11:45:37.538250 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":13,"spinlock_wait_cycles":23168,"thread_start_us":88,"threads_started":1}
I20260806 11:45:37.538801 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.531701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.0.113.1:41095)
0806 11:45:37.533486 (+ 1785us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.533500 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.533525 (+ 25us) server_negotiation.cc:407] Connection header received
0806 11:45:37.533602 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.533607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.533834 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.534243 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:37.534348 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.535424 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.536805 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.537584 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.537696 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.537895 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.538528 (+ 633us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.538632 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1677,"thread_start_us":145,"threads_started":1}
I20260806 11:45:37.540962 26443 tablet_service.cc:1467] Processing CreateTablet for tablet 1353fab9f8b549e4b9a09e4bc54be398 (DEFAULT_TABLE table=test-table [id=5a034b4436fd4412a477cdb9a736b361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:37.541236 26652 tablet_service.cc:1467] Processing CreateTablet for tablet 1353fab9f8b549e4b9a09e4bc54be398 (DEFAULT_TABLE table=test-table [id=5a034b4436fd4412a477cdb9a736b361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:37.541323 26264 tablet_service.cc:1467] Processing CreateTablet for tablet 1353fab9f8b549e4b9a09e4bc54be398 (DEFAULT_TABLE table=test-table [id=5a034b4436fd4412a477cdb9a736b361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:37.542188 26443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1353fab9f8b549e4b9a09e4bc54be398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:37.542230 26264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1353fab9f8b549e4b9a09e4bc54be398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:37.542268 26652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1353fab9f8b549e4b9a09e4bc54be398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:37.549595 26770 tablet_bootstrap.cc:492] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae: Bootstrap starting.
I20260806 11:45:37.549947 26771 tablet_bootstrap.cc:492] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: Bootstrap starting.
I20260806 11:45:37.551540 26770 tablet_bootstrap.cc:654] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:37.551559 26771 tablet_bootstrap.cc:654] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:37.552235 26771 log.cc:824] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:37.552237 26770 log.cc:824] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:37.552667 26769 tablet_bootstrap.cc:492] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: Bootstrap starting.
I20260806 11:45:37.554317 26771 tablet_bootstrap.cc:492] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: No bootstrap required, opened a new log
I20260806 11:45:37.554564 26771 ts_tablet_manager.cc:1397] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 11:45:37.554651 26769 tablet_bootstrap.cc:654] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:37.555207 26770 tablet_bootstrap.cc:492] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae: No bootstrap required, opened a new log
I20260806 11:45:37.555389 26770 ts_tablet_manager.cc:1397] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260806 11:45:37.555524 26769 log.cc:824] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:37.557498 26769 tablet_bootstrap.cc:492] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: No bootstrap required, opened a new log
I20260806 11:45:37.557677 26769 ts_tablet_manager.cc:1397] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:45:37.558182 26771 raft_consensus.cc:348] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } }
I20260806 11:45:37.558426 26771 raft_consensus.cc:374] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:37.558513 26771 raft_consensus.cc:729] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ec6e8ec043436280a4bce1a741c568, State: Initialized, Role: FOLLOWER
I20260806 11:45:37.558992 26771 consensus_queue.cc:260] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } }
I20260806 11:45:37.559216 26770 raft_consensus.cc:348] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } }
I20260806 11:45:37.559643 26550 heartbeater.cc:503] Master 127.0.113.62:37057 was elected leader, sending a full tablet report...
I20260806 11:45:37.559966 26771 ts_tablet_manager.cc:1428] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 11:45:37.560381 26770 raft_consensus.cc:374] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:37.560652 26770 raft_consensus.cc:729] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53be8a5d58746eb8a89c5f1deb120ae, State: Initialized, Role: FOLLOWER
I20260806 11:45:37.561195 26770 consensus_queue.cc:260] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } }
I20260806 11:45:37.561875 26351 heartbeater.cc:503] Master 127.0.113.62:37057 was elected leader, sending a full tablet report...
I20260806 11:45:37.561815 26769 raft_consensus.cc:348] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } }
I20260806 11:45:37.562048 26769 raft_consensus.cc:374] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:37.562095 26769 raft_consensus.cc:729] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87979f5131134c518b2cfe2d337b5412, State: Initialized, Role: FOLLOWER
I20260806 11:45:37.562215 26770 ts_tablet_manager.cc:1428] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260806 11:45:37.562530 26769 consensus_queue.cc:260] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } }
W20260806 11:45:37.563092 26551 tablet.cc:2369] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:45:37.563477 26769 ts_tablet_manager.cc:1428] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260806 11:45:37.563833 26747 heartbeater.cc:503] Master 127.0.113.62:37057 was elected leader, sending a full tablet report...
W20260806 11:45:37.619912 26354 tablet.cc:2369] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:45:37.681658 26781 raft_consensus.cc:482] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:37.681988 26781 raft_consensus.cc:504] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } }
I20260806 11:45:37.683254 26781 leader_election.cc:288] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a53be8a5d58746eb8a89c5f1deb120ae (127.0.113.1:41095), 87979f5131134c518b2cfe2d337b5412 (127.0.113.3:33293)
I20260806 11:45:37.688014 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.683602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:43721 (local address 127.0.113.1:41095)
0806 11:45:37.683735 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.683739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.683754 (+ 15us) server_negotiation.cc:407] Connection header received
0806 11:45:37.683900 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.683906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.683970 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.684059 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:37.684516 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.685294 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.686822 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.687429 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.687540 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.687606 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.687836 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.687914 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 11:45:37.688005 26553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.683486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:41095 (local address 127.0.113.2:43721)
0806 11:45:37.683630 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:37.683647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:37.683745 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:37.684160 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:37.684163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:37.684183 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:37.684390 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.684400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.685430 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.685433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.686694 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.686706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.687551 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.687554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.687633 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.687680 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 11:45:37.687786 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 11:45:37.687871 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 11:45:37.688225 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.684011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:59441 (local address 127.0.113.3:33293)
0806 11:45:37.684136 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.684142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.684290 (+ 148us) server_negotiation.cc:407] Connection header received
0806 11:45:37.684405 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.684408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.684480 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.684568 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:37.684887 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.685571 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.686973 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.687620 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.687772 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.687834 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.688039 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.688110 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 11:45:37.688238 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.683778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:33293 (local address 127.0.113.2:59441)
0806 11:45:37.684190 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:37.684203 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:37.684318 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:37.684586 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:37.684588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:37.684605 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:37.684707 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.684711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.685672 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.685675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.686862 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:37.686870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.687790 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.687794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:37.687909 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.687927 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 11:45:37.688020 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 11:45:37.688090 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":198,"threads_started":1}
I20260806 11:45:37.688832 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1353fab9f8b549e4b9a09e4bc54be398" candidate_uuid: "25ec6e8ec043436280a4bce1a741c568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87979f5131134c518b2cfe2d337b5412" is_pre_election: true
I20260806 11:45:37.688823 26288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1353fab9f8b549e4b9a09e4bc54be398" candidate_uuid: "25ec6e8ec043436280a4bce1a741c568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" is_pre_election: true
I20260806 11:45:37.689138 26288 raft_consensus.cc:2454] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ec6e8ec043436280a4bce1a741c568 in term 0.
I20260806 11:45:37.689147 26677 raft_consensus.cc:2454] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ec6e8ec043436280a4bce1a741c568 in term 0.
I20260806 11:45:37.689656 26388 leader_election.cc:302] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ec6e8ec043436280a4bce1a741c568, a53be8a5d58746eb8a89c5f1deb120ae; no voters:
I20260806 11:45:37.689908 26781 raft_consensus.cc:2781] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:45:37.689986 26781 raft_consensus.cc:482] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:45:37.690016 26781 raft_consensus.cc:3037] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:37.691048 26781 raft_consensus.cc:504] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } }
I20260806 11:45:37.691452 26781 leader_election.cc:288] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [CANDIDATE]: Term 1 election: Requested vote from peers a53be8a5d58746eb8a89c5f1deb120ae (127.0.113.1:41095), 87979f5131134c518b2cfe2d337b5412 (127.0.113.3:33293)
I20260806 11:45:37.691836 26288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1353fab9f8b549e4b9a09e4bc54be398" candidate_uuid: "25ec6e8ec043436280a4bce1a741c568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53be8a5d58746eb8a89c5f1deb120ae"
I20260806 11:45:37.692025 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1353fab9f8b549e4b9a09e4bc54be398" candidate_uuid: "25ec6e8ec043436280a4bce1a741c568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87979f5131134c518b2cfe2d337b5412"
I20260806 11:45:37.692026 26288 raft_consensus.cc:3037] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:37.692144 26677 raft_consensus.cc:3037] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:37.693389 26677 raft_consensus.cc:2454] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ec6e8ec043436280a4bce1a741c568 in term 1.
I20260806 11:45:37.693389 26288 raft_consensus.cc:2454] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ec6e8ec043436280a4bce1a741c568 in term 1.
I20260806 11:45:37.693701 26388 leader_election.cc:302] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ec6e8ec043436280a4bce1a741c568, a53be8a5d58746eb8a89c5f1deb120ae; no voters:
I20260806 11:45:37.693873 26781 raft_consensus.cc:2781] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:37.694097 26781 raft_consensus.cc:686] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 1 LEADER]: Becoming Leader. State: Replica: 25ec6e8ec043436280a4bce1a741c568, State: Running, Role: LEADER
I20260806 11:45:37.694432 26781 consensus_queue.cc:237] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } }
I20260806 11:45:37.697582 26090 catalog_manager.cc:5495] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25ec6e8ec043436280a4bce1a741c568 (127.0.113.2). New cstate: current_term: 1 leader_uuid: "25ec6e8ec043436280a4bce1a741c568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ec6e8ec043436280a4bce1a741c568" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 42643 } health_report { overall_health: HEALTHY } } }
I20260806 11:45:37.727522 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.723836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.0.113.2:42643)
0806 11:45:37.723997 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:37.724002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:37.724017 (+ 15us) server_negotiation.cc:407] Connection header received
0806 11:45:37.724133 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:37.724136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:37.724194 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:37.724360 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:45:37.724618 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.725116 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.725971 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:37.726210 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:37.726324 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:37.726645 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:45:37.727079 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:45:37.727211 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:37.727285 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 11:45:37.727345 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 11:45:37.733481 26677 raft_consensus.cc:1261] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [term 1 FOLLOWER]: Refusing update from remote peer 25ec6e8ec043436280a4bce1a741c568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:37.733481 26288 raft_consensus.cc:1261] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [term 1 FOLLOWER]: Refusing update from remote peer 25ec6e8ec043436280a4bce1a741c568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:37.734230 26781 consensus_queue.cc:1035] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87979f5131134c518b2cfe2d337b5412" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:37.734503 26786 consensus_queue.cc:1035] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a53be8a5d58746eb8a89c5f1deb120ae" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:37.741979 26790 mvcc.cc:205] Tried to move back new op lower bound from 7315524557748412416 to 7315524557599055872. Current Snapshot: MvccSnapshot[applied={T|T < 7315524557748412416}]
I20260806 11:45:37.742226 26792 mvcc.cc:205] Tried to move back new op lower bound from 7315524557748412416 to 7315524557599055872. Current Snapshot: MvccSnapshot[applied={T|T < 7315524557748412416}]
I20260806 11:45:37.744478 26794 mvcc.cc:205] Tried to move back new op lower bound from 7315524557748412416 to 7315524557599055872. Current Snapshot: MvccSnapshot[applied={T|T < 7315524557748412416}]
I20260806 11:45:37.751235 26091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:37.751410 26091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:37.753863 26091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc04ec8685be4f1183154a01a42918d3: Sending DeleteTablet for 3 replicas of tablet 1353fab9f8b549e4b9a09e4bc54be398
I20260806 11:45:37.754688 26443 tablet_service.cc:1514] Processing DeleteTablet for tablet 1353fab9f8b549e4b9a09e4bc54be398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:37 UTC) from {username='slave'} at 127.0.0.1:55976
I20260806 11:45:37.754691 26652 tablet_service.cc:1514] Processing DeleteTablet for tablet 1353fab9f8b549e4b9a09e4bc54be398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:37 UTC) from {username='slave'} at 127.0.0.1:33642
I20260806 11:45:37.754709 26264 tablet_service.cc:1514] Processing DeleteTablet for tablet 1353fab9f8b549e4b9a09e4bc54be398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:37 UTC) from {username='slave'} at 127.0.0.1:54748
I20260806 11:45:37.755381 26805 tablet_replica.cc:330] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: stopping tablet replica
I20260806 11:45:37.755400 26808 tablet_replica.cc:330] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae: stopping tablet replica
I20260806 11:45:37.755709 26805 raft_consensus.cc:2229] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:45:37.755712 26808 raft_consensus.cc:2229] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:45:37.756068 26805 raft_consensus.cc:2258] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:37.756088 26808 raft_consensus.cc:2258] T 1353fab9f8b549e4b9a09e4bc54be398 P a53be8a5d58746eb8a89c5f1deb120ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:37.756212 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 26176
I20260806 11:45:37.756460 26807 tablet_replica.cc:330] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: stopping tablet replica
I20260806 11:45:37.756705 26807 raft_consensus.cc:2229] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:45:37.756986 26807 raft_consensus.cc:2258] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:37.759349 26805 ts_tablet_manager.cc:1905] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:37.762681 26805 ts_tablet_manager.cc:1918] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:37.762785 26805 log.cc:1196] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/1353fab9f8b549e4b9a09e4bc54be398
I20260806 11:45:37.763060 26805 ts_tablet_manager.cc:1939] T 1353fab9f8b549e4b9a09e4bc54be398 P 25ec6e8ec043436280a4bce1a741c568: Deleting consensus metadata
I20260806 11:45:37.764526 26807 ts_tablet_manager.cc:1905] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:37.764873 26067 catalog_manager.cc:4841] TS 25ec6e8ec043436280a4bce1a741c568 (127.0.113.2:42643): tablet 1353fab9f8b549e4b9a09e4bc54be398 (table test-table [id=5a034b4436fd4412a477cdb9a736b361]) successfully deleted
I20260806 11:45:37.767238 26807 ts_tablet_manager.cc:1918] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:37.767343 26807 log.cc:1196] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/1353fab9f8b549e4b9a09e4bc54be398
I20260806 11:45:37.767683 26807 ts_tablet_manager.cc:1939] T 1353fab9f8b549e4b9a09e4bc54be398 P 87979f5131134c518b2cfe2d337b5412: Deleting consensus metadata
I20260806 11:45:37.768985 26070 catalog_manager.cc:4841] TS 87979f5131134c518b2cfe2d337b5412 (127.0.113.3:33293): tablet 1353fab9f8b549e4b9a09e4bc54be398 (table test-table [id=5a034b4436fd4412a477cdb9a736b361]) successfully deleted
W20260806 11:45:37.770555 26070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:41095 (error 108)
I20260806 11:45:37.770759 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 26360
I20260806 11:45:37.776120 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:37.774271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:41095 (local address 127.0.0.1:54752)
0806 11:45:37.775912 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:37.776008 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:41095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1522}
W20260806 11:45:37.776705 26070 catalog_manager.cc:4584] TS a53be8a5d58746eb8a89c5f1deb120ae (127.0.113.1:41095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1353fab9f8b549e4b9a09e4bc54be398: Network error: Client connection negotiation failed: client connection to 127.0.113.1:41095: connect: Connection refused (error 111)
I20260806 11:45:37.783125 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 26557
I20260806 11:45:37.794239 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 26044
2026-08-06T11:45:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1155 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:37 dist-test-slave-45dx krb5kdc[26812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:37 dist-test-slave-45dx krb5kdc[26812](info): set up 2 sockets
Aug 06 11:45:37 dist-test-slave-45dx krb5kdc[26812](info): commencing operation
krb5kdc: starting...
W20260806 11:45:39.123869 452 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.007s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:39 dist-test-slave-45dx krb5kdc[26812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:45:39.260119 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:41305
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:41305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:39.373874 26833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:39.374190 26833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:39.374231 26833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:39.378275 26833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:39.378333 26833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:39.378352 26833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:39.378369 26833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:39.378384 26833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:39.383713 26833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:41305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:41305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:39.384821 26833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:39.385936 26833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:39.394550 26840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:39.394593 26843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:39.394551 26841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:39.394769 26833 server_base.cc:1034] running on GCE node
I20260806 11:45:39.395500 26833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:39.396586 26833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:39.397789 26833 hybrid_clock.cc:648] HybridClock initialized: now 1786016739397776 us; error 31 us; skew 500 ppm
Aug 06 11:45:39 dist-test-slave-45dx krb5kdc[26812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:39.412895 26833 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:45:39.413872 26833 webserver.cc:460] Webserver started at http://127.0.113.62:37201/ using document root <none> and password file <none>
I20260806 11:45:39.414458 26833 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:39.414553 26833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:39.414793 26833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:39.416931 26833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "1784e99e953b47c4bfcfdaf7c6c4162c"
format_stamp: "Formatted at 2026-08-06 11:45:39 on dist-test-slave-45dx"
server_key: "fc2bf959102779e8431191a59c4721e5"
server_key_iv: "3dff9f4a6f4c67c2de0b10bd2684aca5"
server_key_version: "encryptionkey@0"
I20260806 11:45:39.417533 26833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "1784e99e953b47c4bfcfdaf7c6c4162c"
format_stamp: "Formatted at 2026-08-06 11:45:39 on dist-test-slave-45dx"
server_key: "fc2bf959102779e8431191a59c4721e5"
server_key_iv: "3dff9f4a6f4c67c2de0b10bd2684aca5"
server_key_version: "encryptionkey@0"
I20260806 11:45:39.421439 26833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 11:45:39.423925 26850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:39.425209 26833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:45:39.425472 26833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "1784e99e953b47c4bfcfdaf7c6c4162c"
format_stamp: "Formatted at 2026-08-06 11:45:39 on dist-test-slave-45dx"
server_key: "fc2bf959102779e8431191a59c4721e5"
server_key_iv: "3dff9f4a6f4c67c2de0b10bd2684aca5"
server_key_version: "encryptionkey@0"
I20260806 11:45:39.425604 26833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:39.440910 26833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:39.443996 26833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:39.444222 26833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:39.451694 26833 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:41305
I20260806 11:45:39.451720 26925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:41305 every 8 connection(s)
I20260806 11:45:39.452680 26833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:39.455428 26926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:39.457901 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 26833
I20260806 11:45:39.458060 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:39.458396 452 external_mini_cluster.cc:1428] Setting key d601d3733a0d53c2693bbb8fb66d0bcf
I20260806 11:45:39.461885 26926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c: Bootstrap starting.
I20260806 11:45:39.464053 26926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:39.464866 26926 log.cc:824] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:39.467171 26926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c: No bootstrap required, opened a new log
Aug 06 11:45:39 dist-test-slave-45dx krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:45:39.470480 26926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1784e99e953b47c4bfcfdaf7c6c4162c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41305 } }
I20260806 11:45:39.470721 26926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:39.470803 26926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1784e99e953b47c4bfcfdaf7c6c4162c, State: Initialized, Role: FOLLOWER
I20260806 11:45:39.471325 26926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1784e99e953b47c4bfcfdaf7c6c4162c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41305 } }
I20260806 11:45:39.471537 26926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:39.471606 26926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:39.471702 26926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:39.472760 26926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1784e99e953b47c4bfcfdaf7c6c4162c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41305 } }
I20260806 11:45:39.473145 26926 leader_election.cc:302] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1784e99e953b47c4bfcfdaf7c6c4162c; no voters:
I20260806 11:45:39.473502 26926 leader_election.cc:288] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:39.473570 26932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:39.473770 26932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [term 1 LEADER]: Becoming Leader. State: Replica: 1784e99e953b47c4bfcfdaf7c6c4162c, State: Running, Role: LEADER
I20260806 11:45:39.474086 26932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1784e99e953b47c4bfcfdaf7c6c4162c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41305 } }
I20260806 11:45:39.474318 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:39.460063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.0.113.62:41305)
0806 11:45:39.460670 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:39.460685 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:39.460719 (+ 34us) server_negotiation.cc:407] Connection header received
0806 11:45:39.461330 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:39.461349 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:39.461659 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:39.462026 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:39.462883 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.463875 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.464880 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.465523 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.465695 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:39.468582 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:39.468587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:39.468599 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:39.468626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:39.470896 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:39.471423 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:39.471428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:39.471431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:39.471591 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:39.471941 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:39.471943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:39.471944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:39.472393 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:39.472554 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:39.472899 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 11:45:39.473286 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":153,"threads_started":1}
I20260806 11:45:39.474754 26926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:39.475766 26936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1784e99e953b47c4bfcfdaf7c6c4162c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1784e99e953b47c4bfcfdaf7c6c4162c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41305 } } }
I20260806 11:45:39.475889 26936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:39.476202 26937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1784e99e953b47c4bfcfdaf7c6c4162c. Latest consensus state: current_term: 1 leader_uuid: "1784e99e953b47c4bfcfdaf7c6c4162c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1784e99e953b47c4bfcfdaf7c6c4162c" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41305 } } }
I20260806 11:45:39.476330 26937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:39.477083 26940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:39.479849 26940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:39.485400 26940 catalog_manager.cc:1347] Generated new cluster ID: 7eb7787a76a746c787aee4d9270d7c44
I20260806 11:45:39.485464 26940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:39.498595 26940 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:39.499442 26940 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:39.506946 26940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c: Generated new TSK 0
I20260806 11:45:39.507658 26940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:45:39.548244 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41305
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:39.660967 26963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:39.661226 26963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:39.661265 26963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:39.665127 26963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:39.665174 26963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:39.665251 26963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:45:39.670293 26963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:39.671319 26963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:39.672362 26963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:39.682260 26972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:39.682325 26963 server_base.cc:1034] running on GCE node
W20260806 11:45:39.682267 26971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:39.682267 26978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:39.682996 26963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:39.683590 26963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:39.684769 26963 hybrid_clock.cc:648] HybridClock initialized: now 1786016739684742 us; error 31 us; skew 500 ppm
Aug 06 11:45:39 dist-test-slave-45dx krb5kdc[26812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:39.687685 26963 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:45:39.688597 26963 webserver.cc:460] Webserver started at http://127.0.113.1:41707/ using document root <none> and password file <none>
I20260806 11:45:39.689122 26963 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:39.689179 26963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:39.689343 26963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:39.691366 26963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "8a20729b80ff4e64a70566bd65c4af52"
format_stamp: "Formatted at 2026-08-06 11:45:39 on dist-test-slave-45dx"
server_key: "fcac8deb11b5038328a320596c477949"
server_key_iv: "af451c1579524dc60467cc81fe6c0855"
server_key_version: "encryptionkey@0"
I20260806 11:45:39.691954 26963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "8a20729b80ff4e64a70566bd65c4af52"
format_stamp: "Formatted at 2026-08-06 11:45:39 on dist-test-slave-45dx"
server_key: "fcac8deb11b5038328a320596c477949"
server_key_iv: "af451c1579524dc60467cc81fe6c0855"
server_key_version: "encryptionkey@0"
I20260806 11:45:39.695719 26963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 11:45:39.698225 26986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:39.699388 26963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:39.699525 26963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "8a20729b80ff4e64a70566bd65c4af52"
format_stamp: "Formatted at 2026-08-06 11:45:39 on dist-test-slave-45dx"
server_key: "fcac8deb11b5038328a320596c477949"
server_key_iv: "af451c1579524dc60467cc81fe6c0855"
server_key_version: "encryptionkey@0"
I20260806 11:45:39.699636 26963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:39.718093 26963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:39.721258 26963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:39.721495 26963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:39.722133 26963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:39.722988 26963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:39.723037 26963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:39.723114 26963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:39.723146 26963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:39.732667 26963 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:43571
I20260806 11:45:39.732724 27147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:43571 every 8 connection(s)
I20260806 11:45:39.733619 26963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:45:39.735447 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 26963
I20260806 11:45:39.735595 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:45:39.735929 452 external_mini_cluster.cc:1428] Setting key d686a7c13b9f29a902890a73466d5363
Aug 06 11:45:39 dist-test-slave-45dx krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:45:39.748068 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:39.735516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:35259 (local address 127.0.113.62:41305)
0806 11:45:39.735688 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:39.735692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:39.736712 (+ 1020us) server_negotiation.cc:407] Connection header received
0806 11:45:39.737742 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:39.737745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:39.737789 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:39.737858 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:39.739499 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.740025 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.741290 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.741536 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.741657 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:39.744335 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:39.744339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:39.744342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:39.744375 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:39.745862 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:39.746405 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:39.746409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:39.746411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:39.746504 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:39.746853 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:39.746858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:39.746860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:39.747090 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:39.747195 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:39.747754 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 11:45:39.747890 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 11:45:39.749038 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:39.735945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:41305 (local address 127.0.113.1:35259)
0806 11:45:39.736516 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:39.736562 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:39.737528 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:39.738044 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:39.738055 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:39.738609 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:39.739328 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:39.739352 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.740163 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.740166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:39.741157 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:39.741168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.741705 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.741709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:39.741907 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:39.742567 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:39.742584 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:39.744139 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:39.745996 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:39.746003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:39.746020 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:39.746297 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:39.746599 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:39.746603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:39.746605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:39.746756 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:39.747209 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:39.747218 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:39.747511 (+ 293us) client_negotiation.cc:769] Sending connection context
0806 11:45:39.747712 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 11:45:39.748084 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:45:39.750344 27148 heartbeater.cc:347] Connected to a master server at 127.0.113.62:41305
I20260806 11:45:39.752351 27148 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:39.752902 27148 heartbeater.cc:511] Master 127.0.113.62:41305 requested a full tablet report, sending...
I20260806 11:45:39.754565 26873 ts_manager.cc:194] Registered new tserver with Master: 8a20729b80ff4e64a70566bd65c4af52 (127.0.113.1:43571)
I20260806 11:45:39.755721 26873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:35259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:45:39.767719 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41305
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:39.887326 27156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:39.887583 27156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:39.887625 27156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:39.891716 27156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:39.891776 27156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:39.891868 27156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:45:39.897320 27156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:39.898355 27156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:39.899386 27156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 11:45:39.909557 27156 server_base.cc:1034] running on GCE node
W20260806 11:45:39.909514 27162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:39.909519 27165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:39.909578 27163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:39.910158 27156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:39.910722 27156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:39.911890 27156 hybrid_clock.cc:648] HybridClock initialized: now 1786016739911869 us; error 34 us; skew 500 ppm
Aug 06 11:45:39 dist-test-slave-45dx krb5kdc[26812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:39.914685 27156 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:45:39.915549 27156 webserver.cc:460] Webserver started at http://127.0.113.2:40153/ using document root <none> and password file <none>
I20260806 11:45:39.916074 27156 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:39.916134 27156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:39.916294 27156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:39.918367 27156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "ce042f65ff5049318ab7ea4420b19f54"
format_stamp: "Formatted at 2026-08-06 11:45:39 on dist-test-slave-45dx"
server_key: "5177623e0bf194795be2eca2b55b8393"
server_key_iv: "ec8f64e34573980104b9fa9fec8dd4cc"
server_key_version: "encryptionkey@0"
I20260806 11:45:39.918895 27156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "ce042f65ff5049318ab7ea4420b19f54"
format_stamp: "Formatted at 2026-08-06 11:45:39 on dist-test-slave-45dx"
server_key: "5177623e0bf194795be2eca2b55b8393"
server_key_iv: "ec8f64e34573980104b9fa9fec8dd4cc"
server_key_version: "encryptionkey@0"
I20260806 11:45:39.922591 27156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 11:45:39.925027 27179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:39.926308 27156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:39.926416 27156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "ce042f65ff5049318ab7ea4420b19f54"
format_stamp: "Formatted at 2026-08-06 11:45:39 on dist-test-slave-45dx"
server_key: "5177623e0bf194795be2eca2b55b8393"
server_key_iv: "ec8f64e34573980104b9fa9fec8dd4cc"
server_key_version: "encryptionkey@0"
I20260806 11:45:39.926504 27156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:39.945557 27156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:39.948760 27156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:39.948921 27156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:39.949493 27156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:39.950275 27156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:39.950323 27156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:39.950371 27156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:39.950392 27156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:39.959936 27156 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:40091
I20260806 11:45:39.959970 27336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:40091 every 8 connection(s)
I20260806 11:45:39.960867 27156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:45:39.965618 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 27156
I20260806 11:45:39.965849 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:45:39.966200 452 external_mini_cluster.cc:1428] Setting key 7b5d481421dbbe5371c8c6889f71a9b9
Aug 06 11:45:39 dist-test-slave-45dx krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:45:39.975586 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:39.962764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:59069 (local address 127.0.113.62:41305)
0806 11:45:39.962918 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:39.962922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:39.963720 (+ 798us) server_negotiation.cc:407] Connection header received
0806 11:45:39.964714 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:39.964718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:39.964800 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:39.964899 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:39.966215 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.966901 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.968180 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.968506 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.968621 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:39.971634 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:39.971638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:39.971640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:39.971665 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:39.973187 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:39.973875 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:39.973878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:39.973879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:39.973960 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:39.974380 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:39.974383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:39.974384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:39.974554 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:39.974630 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:39.975365 (+ 735us) server_negotiation.cc:299] Negotiation successful
0806 11:45:39.975464 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 11:45:39.976868 27343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:39.963083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:41305 (local address 127.0.113.2:59069)
0806 11:45:39.963555 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:39.963598 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:39.964498 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:39.965084 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:39.965091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:39.965468 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:39.966061 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:39.966076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.967030 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.967033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:39.968046 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:39.968057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:39.968664 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:39.968668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:39.968836 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:39.969450 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:39.969467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:39.971445 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:39.973328 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:39.973335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:39.973350 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:39.973745 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:39.974076 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:39.974080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:39.974082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:39.974278 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:39.974667 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:39.974676 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:39.975047 (+ 371us) client_negotiation.cc:769] Sending connection context
0806 11:45:39.975319 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 11:45:39.975700 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260806 11:45:39.978152 27337 heartbeater.cc:347] Connected to a master server at 127.0.113.62:41305
I20260806 11:45:39.978401 27337 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
I20260806 11:45:39.978943 27337 heartbeater.cc:511] Master 127.0.113.62:41305 requested a full tablet report, sending...
I20260806 11:45:39.980031 26873 ts_manager.cc:194] Registered new tserver with Master: ce042f65ff5049318ab7ea4420b19f54 (127.0.113.2:40091)
I20260806 11:45:39.980700 26873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:59069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:45:39.999541 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41305
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:40.113366 27347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:40.113643 27347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:40.113687 27347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:40.117727 27347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:40.117792 27347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:40.117851 27347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:45:40.123274 27347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:40.124395 27347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:40.125391 27347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:40.135409 27359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:40.135422 27356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:40.135491 27347 server_base.cc:1034] running on GCE node
W20260806 11:45:40.135461 27357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:40.136085 27347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:40.136812 27347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:40.137982 27347 hybrid_clock.cc:648] HybridClock initialized: now 1786016740137968 us; error 26 us; skew 500 ppm
Aug 06 11:45:40 dist-test-slave-45dx krb5kdc[26812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:40.140818 27347 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:45:40.141672 27347 webserver.cc:460] Webserver started at http://127.0.113.3:42241/ using document root <none> and password file <none>
I20260806 11:45:40.142216 27347 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:40.142278 27347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:40.142459 27347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:40.144578 27347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "8bdfb612a1d04564892613ec84c7a63f"
format_stamp: "Formatted at 2026-08-06 11:45:40 on dist-test-slave-45dx"
server_key: "5d04c657acf67359201fab27fafd91dd"
server_key_iv: "de2faa94f5d703abdc4489419f81cf60"
server_key_version: "encryptionkey@0"
I20260806 11:45:40.145164 27347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "8bdfb612a1d04564892613ec84c7a63f"
format_stamp: "Formatted at 2026-08-06 11:45:40 on dist-test-slave-45dx"
server_key: "5d04c657acf67359201fab27fafd91dd"
server_key_iv: "de2faa94f5d703abdc4489419f81cf60"
server_key_version: "encryptionkey@0"
I20260806 11:45:40.149148 27347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 11:45:40.151665 27369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:40.152982 27347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:45:40.153126 27347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "8bdfb612a1d04564892613ec84c7a63f"
format_stamp: "Formatted at 2026-08-06 11:45:40 on dist-test-slave-45dx"
server_key: "5d04c657acf67359201fab27fafd91dd"
server_key_iv: "de2faa94f5d703abdc4489419f81cf60"
server_key_version: "encryptionkey@0"
I20260806 11:45:40.153249 27347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:40.185245 27347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:40.188964 27347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:40.189193 27347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:40.189851 27347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:40.190747 27347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:40.190822 27347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:40.190883 27347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:40.190938 27347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:40.200359 27347 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:42711
I20260806 11:45:40.200385 27532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:42711 every 8 connection(s)
I20260806 11:45:40.201263 27347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:45:40.207513 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 27347
I20260806 11:45:40.207666 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:45:40.207962 452 external_mini_cluster.cc:1428] Setting key 772eec7d86dc59730a35810dd0d7bbf7
Aug 06 11:45:40 dist-test-slave-45dx krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:45:40.215408 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.203117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:37581 (local address 127.0.113.62:41305)
0806 11:45:40.203273 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.203281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.204542 (+ 1261us) server_negotiation.cc:407] Connection header received
0806 11:45:40.205455 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.205459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.205522 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.205606 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:40.206800 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.207348 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.208752 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.208981 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.209095 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.211632 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:40.211636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:40.211639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:40.211670 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:40.213325 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:40.213842 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:40.213846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:40.213848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:40.213932 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:40.214270 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:40.214275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:40.214277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:40.214469 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:40.214578 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.215083 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.215226 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 11:45:40.216220 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.203548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:41305 (local address 127.0.113.3:37581)
0806 11:45:40.204382 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:40.204434 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:40.205222 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:40.205783 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:40.205789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:40.206159 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:40.206636 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.206649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.207486 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.207489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.208627 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.208637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.209138 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.209142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.209312 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.209975 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:40.209992 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:40.211471 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:40.213448 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:40.213454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:40.213468 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:40.213729 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:40.214025 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:40.214028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:40.214030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:40.214175 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:40.214580 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:40.214589 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:40.214858 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 11:45:40.215043 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 11:45:40.215387 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":115,"threads_started":1}
I20260806 11:45:40.217451 27534 heartbeater.cc:347] Connected to a master server at 127.0.113.62:41305
I20260806 11:45:40.217732 27534 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:40.218294 27534 heartbeater.cc:511] Master 127.0.113.62:41305 requested a full tablet report, sending...
I20260806 11:45:40.219404 26873 ts_manager.cc:194] Registered new tserver with Master: 8bdfb612a1d04564892613ec84c7a63f (127.0.113.3:42711)
I20260806 11:45:40.220032 26873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:37581
I20260806 11:45:40.222011 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:45:40.232609 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.223990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.0.113.62:41305)
0806 11:45:40.224127 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.224131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.224451 (+ 320us) server_negotiation.cc:407] Connection header received
0806 11:45:40.224607 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.224612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.224659 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.224752 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:40.225468 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.225996 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.226866 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.227079 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.227188 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.229030 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:40.229034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:40.229037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:40.229071 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:40.230652 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:40.231200 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:40.231204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:40.231206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:40.231293 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:40.231637 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:40.231640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:40.231642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:40.231857 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:40.232019 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.232294 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.232441 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 11:45:40.235430 26873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:45:40.238554 26873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:45:40.253710 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.247308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.0.113.3:42711)
0806 11:45:40.248730 (+ 1422us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.248742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.248763 (+ 21us) server_negotiation.cc:407] Connection header received
0806 11:45:40.248841 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.248848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.249037 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.249188 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:40.249578 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.250662 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.252039 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.252906 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.253025 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.253213 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.253434 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.253533 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":111,"threads_started":1}
I20260806 11:45:40.253713 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.247997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:43571 (local address 127.0.0.1:59908)
0806 11:45:40.248329 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:40.248343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:40.248443 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:40.249228 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:40.249231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:40.249240 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:40.249513 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.249524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.250772 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.250775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.252070 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.252081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.253133 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.253154 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.253250 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.253274 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 11:45:40.253348 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 11:45:40.253545 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":240,"thread_start_us":122,"threads_started":1}
I20260806 11:45:40.253945 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.248276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.0.113.1:43571)
0806 11:45:40.248801 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.248818 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.248854 (+ 36us) server_negotiation.cc:407] Connection header received
0806 11:45:40.248930 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.248934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.249076 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.249432 (+ 356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:40.249601 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.250662 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.252195 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.253004 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.253495 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.253632 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.253679 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.253779 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":118,"threads_started":1}
I20260806 11:45:40.253713 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.247143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:42711 (local address 127.0.0.1:35314)
0806 11:45:40.247541 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:40.247593 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:40.247713 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:40.249274 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:40.249276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:40.249301 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:40.249443 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.249450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.250812 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.250815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.251912 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.251922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.253045 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.253049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.253221 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.253267 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 11:45:40.253384 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 11:45:40.253545 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":52,"thread_start_us":98,"threads_started":1}
I20260806 11:45:40.254122 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.247362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:40091 (local address 127.0.0.1:46764)
0806 11:45:40.247999 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:40.248013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:40.248112 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:40.249205 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:40.249213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:40.249237 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:40.249674 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.249679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.250586 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.250591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.252070 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.252081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.253095 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.253098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.253808 (+ 710us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.253847 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 11:45:40.253924 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 11:45:40.253996 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":954,"thread_start_us":101,"threads_started":1}
I20260806 11:45:40.254819 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.248290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46764 (local address 127.0.113.2:40091)
0806 11:45:40.248783 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.248806 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.248824 (+ 18us) server_negotiation.cc:407] Connection header received
0806 11:45:40.248883 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.248887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.249036 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.249188 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:40.249784 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.250462 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.252418 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.252944 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.253216 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.253388 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.254560 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.254655 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":145,"threads_started":1}
I20260806 11:45:40.256850 27054 tablet_service.cc:1467] Processing CreateTablet for tablet 7c53e32881b7488c84f496b5b4ad5d18 (DEFAULT_TABLE table=test-table [id=8d3a3a3a06df44d5865dfb634e0575b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:40.256840 27435 tablet_service.cc:1467] Processing CreateTablet for tablet 7c53e32881b7488c84f496b5b4ad5d18 (DEFAULT_TABLE table=test-table [id=8d3a3a3a06df44d5865dfb634e0575b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:40.257851 27435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c53e32881b7488c84f496b5b4ad5d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:40.257856 27054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c53e32881b7488c84f496b5b4ad5d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:40.265411 27556 tablet_bootstrap.cc:492] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: Bootstrap starting.
I20260806 11:45:40.267692 27240 tablet_service.cc:1467] Processing CreateTablet for tablet 7c53e32881b7488c84f496b5b4ad5d18 (DEFAULT_TABLE table=test-table [id=8d3a3a3a06df44d5865dfb634e0575b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:40.268551 27556 tablet_bootstrap.cc:654] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:40.268884 27240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c53e32881b7488c84f496b5b4ad5d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:40.269853 27556 log.cc:824] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:40.272073 27556 tablet_bootstrap.cc:492] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: No bootstrap required, opened a new log
I20260806 11:45:40.272280 27556 ts_tablet_manager.cc:1397] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 11:45:40.274122 27560 tablet_bootstrap.cc:492] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52: Bootstrap starting.
I20260806 11:45:40.276285 27560 tablet_bootstrap.cc:654] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:40.276396 27561 tablet_bootstrap.cc:492] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: Bootstrap starting.
I20260806 11:45:40.276288 27556 raft_consensus.cc:348] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } }
I20260806 11:45:40.276619 27556 raft_consensus.cc:374] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:40.276685 27556 raft_consensus.cc:729] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bdfb612a1d04564892613ec84c7a63f, State: Initialized, Role: FOLLOWER
I20260806 11:45:40.277148 27556 consensus_queue.cc:260] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } }
I20260806 11:45:40.277246 27560 log.cc:824] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:40.278128 27534 heartbeater.cc:503] Master 127.0.113.62:41305 was elected leader, sending a full tablet report...
I20260806 11:45:40.278191 27556 ts_tablet_manager.cc:1428] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260806 11:45:40.279381 27560 tablet_bootstrap.cc:492] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52: No bootstrap required, opened a new log
I20260806 11:45:40.279575 27560 ts_tablet_manager.cc:1397] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260806 11:45:40.280439 27561 tablet_bootstrap.cc:654] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:40.281296 27561 log.cc:824] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:40.283191 27561 tablet_bootstrap.cc:492] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: No bootstrap required, opened a new log
I20260806 11:45:40.283390 27561 ts_tablet_manager.cc:1397] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 11:45:40.283833 27560 raft_consensus.cc:348] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } }
I20260806 11:45:40.284032 27560 raft_consensus.cc:374] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:40.284086 27560 raft_consensus.cc:729] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a20729b80ff4e64a70566bd65c4af52, State: Initialized, Role: FOLLOWER
I20260806 11:45:40.284546 27560 consensus_queue.cc:260] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } }
I20260806 11:45:40.285211 27148 heartbeater.cc:503] Master 127.0.113.62:41305 was elected leader, sending a full tablet report...
I20260806 11:45:40.285478 27560 ts_tablet_manager.cc:1428] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 11:45:40.286883 27561 raft_consensus.cc:348] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } }
I20260806 11:45:40.287155 27561 raft_consensus.cc:374] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:40.287221 27561 raft_consensus.cc:729] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce042f65ff5049318ab7ea4420b19f54, State: Initialized, Role: FOLLOWER
I20260806 11:45:40.287664 27561 consensus_queue.cc:260] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } }
I20260806 11:45:40.288431 27337 heartbeater.cc:503] Master 127.0.113.62:41305 was elected leader, sending a full tablet report...
I20260806 11:45:40.288728 27561 ts_tablet_manager.cc:1428] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 11:45:40.338629 27562 raft_consensus.cc:482] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:40.338848 27562 raft_consensus.cc:504] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } }
I20260806 11:45:40.340022 27562 leader_election.cc:288] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce042f65ff5049318ab7ea4420b19f54 (127.0.113.2:40091), 8a20729b80ff4e64a70566bd65c4af52 (127.0.113.1:43571)
I20260806 11:45:40.344008 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.340354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:41775 (local address 127.0.113.2:40091)
0806 11:45:40.340503 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.340509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.340678 (+ 169us) server_negotiation.cc:407] Connection header received
0806 11:45:40.340751 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.340754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.340835 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.340920 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:40.341241 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.341839 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.342977 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.343406 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.343488 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.343571 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.343797 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.343876 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:45:40.344003 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.340248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:40091 (local address 127.0.113.3:41775)
0806 11:45:40.340501 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:40.340517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:40.340654 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:40.340951 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:40.340954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:40.340976 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:40.341114 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.341125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.342007 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.342011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.342828 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.342846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.343536 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.343543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.343648 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.343674 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 11:45:40.343745 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 11:45:40.343831 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260806 11:45:40.344034 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.340521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:51831 (local address 127.0.113.1:43571)
0806 11:45:40.340673 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.340679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.340922 (+ 243us) server_negotiation.cc:407] Connection header received
0806 11:45:40.341008 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.341011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.341058 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.341144 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:40.341436 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.341995 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.343030 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.343446 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.343577 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.343643 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.343847 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.343928 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 11:45:40.344023 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.340361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:43571 (local address 127.0.113.3:51831)
0806 11:45:40.340779 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:40.340794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:40.340924 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:40.341167 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:40.341169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:40.341211 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:40.341324 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.341331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.342108 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.342110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.342880 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.342888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.343611 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.343619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.343698 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.343716 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 11:45:40.343815 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 11:45:40.343889 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":152,"threads_started":1}
I20260806 11:45:40.344904 27087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c53e32881b7488c84f496b5b4ad5d18" candidate_uuid: "8bdfb612a1d04564892613ec84c7a63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a20729b80ff4e64a70566bd65c4af52" is_pre_election: true
I20260806 11:45:40.344892 27269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c53e32881b7488c84f496b5b4ad5d18" candidate_uuid: "8bdfb612a1d04564892613ec84c7a63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce042f65ff5049318ab7ea4420b19f54" is_pre_election: true
I20260806 11:45:40.345218 27087 raft_consensus.cc:2454] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bdfb612a1d04564892613ec84c7a63f in term 0.
I20260806 11:45:40.345218 27269 raft_consensus.cc:2454] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bdfb612a1d04564892613ec84c7a63f in term 0.
I20260806 11:45:40.345742 27373 leader_election.cc:302] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bdfb612a1d04564892613ec84c7a63f, ce042f65ff5049318ab7ea4420b19f54; no voters:
I20260806 11:45:40.346006 27562 raft_consensus.cc:2781] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:45:40.346091 27562 raft_consensus.cc:482] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:45:40.346122 27562 raft_consensus.cc:3037] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:40.347530 27562 raft_consensus.cc:504] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } }
I20260806 11:45:40.347894 27562 leader_election.cc:288] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [CANDIDATE]: Term 1 election: Requested vote from peers ce042f65ff5049318ab7ea4420b19f54 (127.0.113.2:40091), 8a20729b80ff4e64a70566bd65c4af52 (127.0.113.1:43571)
I20260806 11:45:40.348294 27269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c53e32881b7488c84f496b5b4ad5d18" candidate_uuid: "8bdfb612a1d04564892613ec84c7a63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce042f65ff5049318ab7ea4420b19f54"
I20260806 11:45:40.348348 27087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c53e32881b7488c84f496b5b4ad5d18" candidate_uuid: "8bdfb612a1d04564892613ec84c7a63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a20729b80ff4e64a70566bd65c4af52"
I20260806 11:45:40.348475 27269 raft_consensus.cc:3037] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:40.348497 27087 raft_consensus.cc:3037] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:40.349541 27087 raft_consensus.cc:2454] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bdfb612a1d04564892613ec84c7a63f in term 1.
I20260806 11:45:40.349599 27269 raft_consensus.cc:2454] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bdfb612a1d04564892613ec84c7a63f in term 1.
I20260806 11:45:40.349888 27370 leader_election.cc:302] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a20729b80ff4e64a70566bd65c4af52, 8bdfb612a1d04564892613ec84c7a63f; no voters:
I20260806 11:45:40.350071 27562 raft_consensus.cc:2781] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:40.350332 27562 raft_consensus.cc:686] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 1 LEADER]: Becoming Leader. State: Replica: 8bdfb612a1d04564892613ec84c7a63f, State: Running, Role: LEADER
I20260806 11:45:40.350680 27562 consensus_queue.cc:237] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } }
I20260806 11:45:40.353860 26873 catalog_manager.cc:5495] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bdfb612a1d04564892613ec84c7a63f (127.0.113.3). New cstate: current_term: 1 leader_uuid: "8bdfb612a1d04564892613ec84c7a63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bdfb612a1d04564892613ec84c7a63f" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 } health_report { overall_health: UNKNOWN } } }
I20260806 11:45:40.379272 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.375387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.0.113.3:42711)
0806 11:45:40.375524 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.375531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.375547 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:45:40.375739 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.375742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.375807 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.375956 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:45:40.376214 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.376810 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.377713 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.377947 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.378063 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.378363 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:45:40.378756 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:45:40.378932 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.379062 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.379106 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:45:40.384976 27087 raft_consensus.cc:1261] T 7c53e32881b7488c84f496b5b4ad5d18 P 8a20729b80ff4e64a70566bd65c4af52 [term 1 FOLLOWER]: Refusing update from remote peer 8bdfb612a1d04564892613ec84c7a63f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:40.384976 27269 raft_consensus.cc:1261] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [term 1 FOLLOWER]: Refusing update from remote peer 8bdfb612a1d04564892613ec84c7a63f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:40.385730 27570 consensus_queue.cc:1035] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a20729b80ff4e64a70566bd65c4af52" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 43571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:40.385959 27562 consensus_queue.cc:1035] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce042f65ff5049318ab7ea4420b19f54" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 40091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:40.403530 26873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:40.403733 26873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:40.406225 26873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1784e99e953b47c4bfcfdaf7c6c4162c: Sending DeleteTablet for 3 replicas of tablet 7c53e32881b7488c84f496b5b4ad5d18
I20260806 11:45:40.407049 27240 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c53e32881b7488c84f496b5b4ad5d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:40 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:46764
I20260806 11:45:40.407107 27054 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c53e32881b7488c84f496b5b4ad5d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:40 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:59908
I20260806 11:45:40.407171 27435 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c53e32881b7488c84f496b5b4ad5d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:40 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:35314
I20260806 11:45:40.407608 27588 tablet_replica.cc:330] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: stopping tablet replica
I20260806 11:45:40.407781 27590 tablet_replica.cc:330] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: stopping tablet replica
I20260806 11:45:40.407958 27588 raft_consensus.cc:2229] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:45:40.408046 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 26963
I20260806 11:45:40.408290 27588 raft_consensus.cc:2258] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:40.411847 27590 raft_consensus.cc:2229] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:45:40.412310 27590 raft_consensus.cc:2258] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:40.412607 27588 ts_tablet_manager.cc:1905] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:40.414592 27590 ts_tablet_manager.cc:1905] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:40.416518 27588 ts_tablet_manager.cc:1918] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:40.416630 27588 log.cc:1196] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/7c53e32881b7488c84f496b5b4ad5d18
I20260806 11:45:40.417116 27588 ts_tablet_manager.cc:1939] T 7c53e32881b7488c84f496b5b4ad5d18 P ce042f65ff5049318ab7ea4420b19f54: Deleting consensus metadata
I20260806 11:45:40.417635 27590 ts_tablet_manager.cc:1918] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:40.417795 27590 log.cc:1196] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/7c53e32881b7488c84f496b5b4ad5d18
I20260806 11:45:40.418182 27590 ts_tablet_manager.cc:1939] T 7c53e32881b7488c84f496b5b4ad5d18 P 8bdfb612a1d04564892613ec84c7a63f: Deleting consensus metadata
I20260806 11:45:40.418587 26853 catalog_manager.cc:4841] TS ce042f65ff5049318ab7ea4420b19f54 (127.0.113.2:40091): tablet 7c53e32881b7488c84f496b5b4ad5d18 (table test-table [id=8d3a3a3a06df44d5865dfb634e0575b2]) successfully deleted
I20260806 11:45:40.419960 26852 catalog_manager.cc:4841] TS 8bdfb612a1d04564892613ec84c7a63f (127.0.113.3:42711): tablet 7c53e32881b7488c84f496b5b4ad5d18 (table test-table [id=8d3a3a3a06df44d5865dfb634e0575b2]) successfully deleted
W20260806 11:45:40.423547 26851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:43571 (error 108)
I20260806 11:45:40.423820 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 27156
I20260806 11:45:40.433898 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.433437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:43571 (local address 127.0.0.1:59916)
0806 11:45:40.433641 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:40.433751 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:43571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260806 11:45:40.434298 26851 catalog_manager.cc:4584] TS 8a20729b80ff4e64a70566bd65c4af52 (127.0.113.1:43571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c53e32881b7488c84f496b5b4ad5d18: Network error: Client connection negotiation failed: client connection to 127.0.113.1:43571: connect: Connection refused (error 111)
I20260806 11:45:40.441183 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 27347
I20260806 11:45:40.456737 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 26833
2026-08-06T11:45:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2661 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T11:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:40Z Disabled control of system clock
I20260806 11:45:40.491873 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:38359
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:38359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 11:45:40.605394 27604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:40.605646 27604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:40.605680 27604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:40.609913 27604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:40.609973 27604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:40.609988 27604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:40.610005 27604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:40.610020 27604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:40.614715 27604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:38359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:38359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:40.615738 27604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:40.617225 27604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:40.625636 27614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:40.625747 27604 server_base.cc:1034] running on GCE node
W20260806 11:45:40.625677 27616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:40.625708 27613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:40.626746 27604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:40.627848 27604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:40.629040 27604 hybrid_clock.cc:648] HybridClock initialized: now 1786016740629012 us; error 39 us; skew 500 ppm
I20260806 11:45:40.630954 27604 webserver.cc:460] Webserver started at http://127.0.113.62:37645/ using document root <none> and password file <none>
I20260806 11:45:40.631541 27604 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:40.631641 27604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:40.631883 27604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:40.633973 27604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "c47f345ec89749be975aee742e85a8d9"
format_stamp: "Formatted at 2026-08-06 11:45:40 on dist-test-slave-45dx"
server_key: "5d5b04350ce46b67d4770e5a32ce6fa7"
server_key_iv: "129b72a080816cc24ba9d06b50318381"
server_key_version: "encryptionkey@0"
I20260806 11:45:40.634558 27604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "c47f345ec89749be975aee742e85a8d9"
format_stamp: "Formatted at 2026-08-06 11:45:40 on dist-test-slave-45dx"
server_key: "5d5b04350ce46b67d4770e5a32ce6fa7"
server_key_iv: "129b72a080816cc24ba9d06b50318381"
server_key_version: "encryptionkey@0"
I20260806 11:45:40.638418 27604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 11:45:40.640867 27625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:40.642174 27604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:45:40.642328 27604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "c47f345ec89749be975aee742e85a8d9"
format_stamp: "Formatted at 2026-08-06 11:45:40 on dist-test-slave-45dx"
server_key: "5d5b04350ce46b67d4770e5a32ce6fa7"
server_key_iv: "129b72a080816cc24ba9d06b50318381"
server_key_version: "encryptionkey@0"
I20260806 11:45:40.642447 27604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:40.663616 27604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:40.664443 27604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:40.664655 27604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:40.671659 27604 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:38359
I20260806 11:45:40.671684 27694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:38359 every 8 connection(s)
I20260806 11:45:40.672683 27604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:40.675746 27696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:40.678224 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 27604
I20260806 11:45:40.678364 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:40.678725 452 external_mini_cluster.cc:1428] Setting key 77712e1f26ce414dfe5d247018e4458d
I20260806 11:45:40.682004 27696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9: Bootstrap starting.
I20260806 11:45:40.684639 27696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:40.685469 27696 log.cc:824] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:40.687889 27696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9: No bootstrap required, opened a new log
I20260806 11:45:40.688674 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.680149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.0.113.62:38359)
0806 11:45:40.680837 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.680853 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.680890 (+ 37us) server_negotiation.cc:407] Connection header received
0806 11:45:40.681523 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.681539 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.681967 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.682295 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:40.682656 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.683614 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.684875 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.685330 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.685454 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.686435 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:40.686445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:40.686465 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:40.686978 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:40.687053 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:40.687085 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:40.687191 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.687396 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.687671 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":217,"threads_started":1}
I20260806 11:45:40.691975 27696 raft_consensus.cc:348] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f345ec89749be975aee742e85a8d9" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38359 } }
I20260806 11:45:40.692188 27696 raft_consensus.cc:374] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:40.692221 27696 raft_consensus.cc:729] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47f345ec89749be975aee742e85a8d9, State: Initialized, Role: FOLLOWER
I20260806 11:45:40.692639 27696 consensus_queue.cc:260] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f345ec89749be975aee742e85a8d9" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38359 } }
I20260806 11:45:40.692760 27696 raft_consensus.cc:388] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:40.692804 27696 raft_consensus.cc:482] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:40.692876 27696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:40.693718 27696 raft_consensus.cc:504] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f345ec89749be975aee742e85a8d9" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38359 } }
I20260806 11:45:40.694005 27696 leader_election.cc:302] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c47f345ec89749be975aee742e85a8d9; no voters:
I20260806 11:45:40.694304 27696 leader_election.cc:288] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:40.694434 27705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:40.694646 27705 raft_consensus.cc:686] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [term 1 LEADER]: Becoming Leader. State: Replica: c47f345ec89749be975aee742e85a8d9, State: Running, Role: LEADER
I20260806 11:45:40.694968 27705 consensus_queue.cc:237] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f345ec89749be975aee742e85a8d9" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38359 } }
I20260806 11:45:40.695559 27696 sys_catalog.cc:564] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:40.696653 27706 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c47f345ec89749be975aee742e85a8d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f345ec89749be975aee742e85a8d9" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38359 } } }
I20260806 11:45:40.696794 27706 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:40.697079 27707 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c47f345ec89749be975aee742e85a8d9. Latest consensus state: current_term: 1 leader_uuid: "c47f345ec89749be975aee742e85a8d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f345ec89749be975aee742e85a8d9" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 38359 } } }
I20260806 11:45:40.697163 27707 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:40.697209 27717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:40.700330 27717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:40.705931 27717 catalog_manager.cc:1347] Generated new cluster ID: 24df45648c2b49b79c93bf88e9ce3302
I20260806 11:45:40.706004 27717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:40.721967 27717 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:40.722895 27717 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:40.729092 27717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9: Generated new TSK 0
I20260806 11:45:40.729805 27717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:40.738651 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38359
--builtin_ntp_servers=127.0.113.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 11:45:40.855376 27730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:40.855659 27730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:40.855697 27730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:40.859742 27730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:40.859802 27730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:40.859859 27730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:45:40.864665 27730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:40.865777 27730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:40.867198 27730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:40.876771 27742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:40.876754 27743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:40.876832 27730 server_base.cc:1034] running on GCE node
W20260806 11:45:40.876760 27745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:40.877405 27730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:40.878053 27730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:40.879238 27730 hybrid_clock.cc:648] HybridClock initialized: now 1786016740879218 us; error 33 us; skew 500 ppm
I20260806 11:45:40.881160 27730 webserver.cc:460] Webserver started at http://127.0.113.1:36569/ using document root <none> and password file <none>
I20260806 11:45:40.881713 27730 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:40.881789 27730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:40.881994 27730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:40.884011 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "454d521eb36d475ca4e7ae4acc4938f7"
format_stamp: "Formatted at 2026-08-06 11:45:40 on dist-test-slave-45dx"
server_key: "d98e8cf2ece744c4bf6c3a0c86d2d81f"
server_key_iv: "958eefd40a867f27543acab3d51d4b8b"
server_key_version: "encryptionkey@0"
I20260806 11:45:40.884606 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "454d521eb36d475ca4e7ae4acc4938f7"
format_stamp: "Formatted at 2026-08-06 11:45:40 on dist-test-slave-45dx"
server_key: "d98e8cf2ece744c4bf6c3a0c86d2d81f"
server_key_iv: "958eefd40a867f27543acab3d51d4b8b"
server_key_version: "encryptionkey@0"
I20260806 11:45:40.888268 27730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 11:45:40.890681 27753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:40.891947 27730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:40.892092 27730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "454d521eb36d475ca4e7ae4acc4938f7"
format_stamp: "Formatted at 2026-08-06 11:45:40 on dist-test-slave-45dx"
server_key: "d98e8cf2ece744c4bf6c3a0c86d2d81f"
server_key_iv: "958eefd40a867f27543acab3d51d4b8b"
server_key_version: "encryptionkey@0"
I20260806 11:45:40.892220 27730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:40.909911 27730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:40.910670 27730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:40.910895 27730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:40.911540 27730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:40.912639 27730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:40.912712 27730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:40.912788 27730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:40.912832 27730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:40.922228 27730 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:45861
I20260806 11:45:40.922251 27909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:45861 every 8 connection(s)
I20260806 11:45:40.923172 27730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:45:40.926795 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 27730
I20260806 11:45:40.926898 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:45:40.927210 452 external_mini_cluster.cc:1428] Setting key f3a4a6d8c6cd6eee95461026acf8f235
I20260806 11:45:40.929384 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38359
--builtin_ntp_servers=127.0.113.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 11:45:40.933023 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.925118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:57925 (local address 127.0.113.62:38359)
0806 11:45:40.925258 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:40.925262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:40.926325 (+ 1063us) server_negotiation.cc:407] Connection header received
0806 11:45:40.927890 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:40.927893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:40.927948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:40.928051 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:40.929208 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.929834 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.930982 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.931201 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.931314 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.932510 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:40.932513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:40.932516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:40.932535 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:40.932556 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:40.932564 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:40.932642 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:40.932822 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 11:45:40.932883 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 11:45:40.933944 27916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:40.925509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:38359 (local address 127.0.113.1:57925)
0806 11:45:40.926158 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:40.926200 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:40.927653 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:40.928241 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:40.928250 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:40.928395 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:40.929057 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.929073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.929966 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.929969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.930841 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:40.930850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:40.931364 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:40.931369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:40.931528 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:40.932192 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:45:40.932194 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:40.932235 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:45:40.932349 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:40.932658 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:40.932670 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:40.932676 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 11:45:40.932812 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 11:45:40.933090 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260806 11:45:40.935408 27910 heartbeater.cc:347] Connected to a master server at 127.0.113.62:38359
I20260806 11:45:40.935683 27910 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:40.936240 27910 heartbeater.cc:511] Master 127.0.113.62:38359 requested a full tablet report, sending...
I20260806 11:45:40.938155 27648 ts_manager.cc:194] Registered new tserver with Master: 454d521eb36d475ca4e7ae4acc4938f7 (127.0.113.1:45861)
I20260806 11:45:40.940003 27648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.1:57925
W20260806 11:45:41.048097 27917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:41.048405 27917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:41.048444 27917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:41.052532 27917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:41.052588 27917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:41.052642 27917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:45:41.057426 27917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:41.058492 27917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:41.059828 27917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:41.069597 27924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:41.069741 27917 server_base.cc:1034] running on GCE node
W20260806 11:45:41.069633 27925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:41.069943 27928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:41.070367 27917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:41.070996 27917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:41.072190 27917 hybrid_clock.cc:648] HybridClock initialized: now 1786016741072178 us; error 30 us; skew 500 ppm
I20260806 11:45:41.074153 27917 webserver.cc:460] Webserver started at http://127.0.113.2:44387/ using document root <none> and password file <none>
I20260806 11:45:41.074751 27917 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:41.074863 27917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:41.075100 27917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:41.077250 27917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "b58a0638d0794de7a6bbb05213c2751c"
format_stamp: "Formatted at 2026-08-06 11:45:41 on dist-test-slave-45dx"
server_key: "bea5546ef5271469144ab3a902d57ddb"
server_key_iv: "f2c9b10fb3fb9bfe5ab3e6609a502f68"
server_key_version: "encryptionkey@0"
I20260806 11:45:41.077853 27917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "b58a0638d0794de7a6bbb05213c2751c"
format_stamp: "Formatted at 2026-08-06 11:45:41 on dist-test-slave-45dx"
server_key: "bea5546ef5271469144ab3a902d57ddb"
server_key_iv: "f2c9b10fb3fb9bfe5ab3e6609a502f68"
server_key_version: "encryptionkey@0"
I20260806 11:45:41.081586 27917 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:45:41.084066 27941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:41.085294 27917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:45:41.085436 27917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "b58a0638d0794de7a6bbb05213c2751c"
format_stamp: "Formatted at 2026-08-06 11:45:41 on dist-test-slave-45dx"
server_key: "bea5546ef5271469144ab3a902d57ddb"
server_key_iv: "f2c9b10fb3fb9bfe5ab3e6609a502f68"
server_key_version: "encryptionkey@0"
I20260806 11:45:41.085548 27917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:41.099233 27917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:41.099920 27917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:41.100126 27917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:41.100802 27917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:41.101751 27917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:41.101850 27917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:41.101933 27917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:41.101986 27917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:41.111016 27917 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:33563
I20260806 11:45:41.111038 28110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:33563 every 8 connection(s)
I20260806 11:45:41.111964 27917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:45:41.117060 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 27917
I20260806 11:45:41.117197 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:45:41.117519 452 external_mini_cluster.cc:1428] Setting key 948f7e44df0d3e433e60998328ff57f1
I20260806 11:45:41.119539 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38359
--builtin_ntp_servers=127.0.113.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 11:45:41.121943 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.113982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:46283 (local address 127.0.113.62:38359)
0806 11:45:41.114120 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:41.114123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:41.115106 (+ 983us) server_negotiation.cc:407] Connection header received
0806 11:45:41.116626 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:41.116630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:41.116692 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:41.116802 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:41.117822 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.118474 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.119807 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.120034 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.120153 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.121473 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:41.121476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:41.121478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:41.121491 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:41.121511 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:41.121519 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:41.121579 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:41.121769 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 11:45:41.121833 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:45:41.123173 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.114277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:38359 (local address 127.0.113.2:46283)
0806 11:45:41.114952 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:41.114993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:41.116410 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:41.116988 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:41.116995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:41.117076 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:41.117676 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.117692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.118607 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.118611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.119630 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.119641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.120196 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.120200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.120449 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.121122 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:45:41.121123 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:41.121164 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:45:41.121255 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:41.121607 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:41.121618 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:41.121625 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 11:45:41.121765 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 11:45:41.122193 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":116,"threads_started":1}
I20260806 11:45:41.124392 28112 heartbeater.cc:347] Connected to a master server at 127.0.113.62:38359
I20260806 11:45:41.124640 28112 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:41.125149 28112 heartbeater.cc:511] Master 127.0.113.62:38359 requested a full tablet report, sending...
I20260806 11:45:41.126243 27648 ts_manager.cc:194] Registered new tserver with Master: b58a0638d0794de7a6bbb05213c2751c (127.0.113.2:33563)
I20260806 11:45:41.127121 27648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.2:46283
W20260806 11:45:41.236579 28116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:41.236840 28116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:41.236905 28116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:41.240909 28116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:41.240960 28116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:41.241047 28116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:45:41.245754 28116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:38359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:41.246835 28116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:41.248286 28116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:41.258736 28124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:41.258764 28129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:41.258730 28125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:41.259608 28116 server_base.cc:1034] running on GCE node
I20260806 11:45:41.259985 28116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:41.260654 28116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:41.261828 28116 hybrid_clock.cc:648] HybridClock initialized: now 1786016741261793 us; error 45 us; skew 500 ppm
I20260806 11:45:41.263765 28116 webserver.cc:460] Webserver started at http://127.0.113.3:45275/ using document root <none> and password file <none>
I20260806 11:45:41.264390 28116 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:41.264461 28116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:41.264638 28116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:41.267097 28116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "0a97059174374bcfa441c1499eeb0333"
format_stamp: "Formatted at 2026-08-06 11:45:41 on dist-test-slave-45dx"
server_key: "33de83c5c4dafacb32eb7e4e2995c1f5"
server_key_iv: "6f93594ba7cbc7ce9bb919a5b9b975e0"
server_key_version: "encryptionkey@0"
I20260806 11:45:41.267685 28116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "0a97059174374bcfa441c1499eeb0333"
format_stamp: "Formatted at 2026-08-06 11:45:41 on dist-test-slave-45dx"
server_key: "33de83c5c4dafacb32eb7e4e2995c1f5"
server_key_iv: "6f93594ba7cbc7ce9bb919a5b9b975e0"
server_key_version: "encryptionkey@0"
I20260806 11:45:41.271389 28116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:45:41.273957 28140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:41.275215 28116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:45:41.275347 28116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "0a97059174374bcfa441c1499eeb0333"
format_stamp: "Formatted at 2026-08-06 11:45:41 on dist-test-slave-45dx"
server_key: "33de83c5c4dafacb32eb7e4e2995c1f5"
server_key_iv: "6f93594ba7cbc7ce9bb919a5b9b975e0"
server_key_version: "encryptionkey@0"
I20260806 11:45:41.275441 28116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:41.289551 28116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:41.290182 28116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:41.290345 28116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:41.290876 28116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:41.291774 28116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:41.291849 28116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:41.291893 28116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:41.291908 28116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:41.301321 28116 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:42417
I20260806 11:45:41.301338 28302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:42417 every 8 connection(s)
I20260806 11:45:41.302244 28116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:45:41.308148 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 28116
I20260806 11:45:41.308283 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:45:41.308617 452 external_mini_cluster.cc:1428] Setting key 19f4a9efeef0d0e118c1546403bfebdf
I20260806 11:45:41.312260 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.304166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:60769 (local address 127.0.113.62:38359)
0806 11:45:41.304341 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:41.304345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:41.305440 (+ 1095us) server_negotiation.cc:407] Connection header received
0806 11:45:41.306776 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:41.306779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:41.306853 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:41.306931 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:41.307931 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.308595 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.310195 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.310521 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.310637 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.311829 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:41.311833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:41.311836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:41.311852 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:41.311873 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:41.311881 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:41.311945 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:41.312090 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 11:45:41.312144 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 11:45:41.313215 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.304602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:38359 (local address 127.0.113.3:60769)
0806 11:45:41.305264 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:41.305305 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:41.306585 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:41.307094 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:41.307100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:41.307180 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:41.307735 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.307758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.308736 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.308740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.310056 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.310067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.310682 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.310685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.310915 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.311574 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:45:41.311576 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:41.311616 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:45:41.311681 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:41.311960 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:41.311967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:41.311971 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 11:45:41.312070 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 11:45:41.312295 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
I20260806 11:45:41.314311 28303 heartbeater.cc:347] Connected to a master server at 127.0.113.62:38359
I20260806 11:45:41.314576 28303 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:41.315112 28303 heartbeater.cc:511] Master 127.0.113.62:38359 requested a full tablet report, sending...
I20260806 11:45:41.316445 27648 ts_manager.cc:194] Registered new tserver with Master: 0a97059174374bcfa441c1499eeb0333 (127.0.113.3:42417)
I20260806 11:45:41.317098 27648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.3:60769
I20260806 11:45:41.322861 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:45:41.328922 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.325121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.0.113.62:38359)
0806 11:45:41.325268 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:41.325272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:41.325460 (+ 188us) server_negotiation.cc:407] Connection header received
0806 11:45:41.325565 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:41.325567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:41.325608 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:41.325673 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:41.325950 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.326450 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.327334 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.327539 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.327616 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.328448 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:41.328451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:41.328453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:41.328466 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:41.328484 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:41.328492 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:41.328597 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:41.328695 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 11:45:41.328728 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 11:45:41.331939 27648 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:45:41.335060 27648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:45:41.349701 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.344291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.0.113.1:45861)
0806 11:45:41.344705 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:41.344719 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:41.345047 (+ 328us) server_negotiation.cc:407] Connection header received
0806 11:45:41.345131 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:41.345139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:41.345313 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:41.345478 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:41.345888 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.346998 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.348426 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.348975 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.349104 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.349312 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:41.349429 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 11:45:41.349528 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":124,"threads_started":1}
I20260806 11:45:41.349820 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.343595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:42417 (local address 127.0.0.1:46400)
0806 11:45:41.344398 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:41.344454 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:41.344590 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:41.345229 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:41.345235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:41.345256 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:41.345423 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.345443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.346786 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.346791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.348268 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.348274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.349377 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.349381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.349474 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.349496 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 11:45:41.349552 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 11:45:41.349624 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":75,"spinlock_wait_cycles":5888,"thread_start_us":116,"threads_started":1}
I20260806 11:45:41.350163 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.343564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.0.113.2:33563)
0806 11:45:41.344967 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:41.344986 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:41.345011 (+ 25us) server_negotiation.cc:407] Connection header received
0806 11:45:41.345086 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:41.345093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:41.345277 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:41.345490 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:41.345630 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.346884 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.348435 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.349223 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.349340 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.349526 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:41.349896 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 11:45:41.349998 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":129,"threads_started":1}
I20260806 11:45:41.349695 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.343832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:45861 (local address 127.0.0.1:52968)
0806 11:45:41.344844 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:41.344857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:41.344949 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:41.345633 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:41.345636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:41.345652 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:41.345746 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.345754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.347115 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.347120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.348277 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.348287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.349116 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.349119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.349224 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.349262 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 11:45:41.349397 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 11:45:41.349529 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":45,"thread_start_us":96,"threads_started":1}
I20260806 11:45:41.350250 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.344220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.0.113.3:42417)
0806 11:45:41.344705 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:41.344721 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:41.344747 (+ 26us) server_negotiation.cc:407] Connection header received
0806 11:45:41.344829 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:41.344834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:41.345063 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:41.345581 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:41.345716 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.346678 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.348435 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.349223 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.349774 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.349958 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:41.350002 (+ 44us) server_negotiation.cc:299] Negotiation successful
0806 11:45:41.350087 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":173,"threads_started":1}
I20260806 11:45:41.352537 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.343403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:33563 (local address 127.0.0.1:38218)
0806 11:45:41.344398 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:41.344454 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:41.344590 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:41.345420 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:41.345423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:41.345445 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:41.345545 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.345552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.346998 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.347001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.348258 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.348270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.349585 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.349589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.349662 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.349685 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 11:45:41.349751 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 11:45:41.349811 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":13,"spinlock_wait_cycles":6784,"thread_start_us":98,"threads_started":1}
I20260806 11:45:41.353127 27819 tablet_service.cc:1467] Processing CreateTablet for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1 (DEFAULT_TABLE table=test-table [id=74fb7c01183a4b34aa3ff9d5157501d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:41.353116 28209 tablet_service.cc:1467] Processing CreateTablet for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1 (DEFAULT_TABLE table=test-table [id=74fb7c01183a4b34aa3ff9d5157501d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:41.354149 28209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:41.354148 27819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:41.360960 28327 tablet_bootstrap.cc:492] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7: Bootstrap starting.
I20260806 11:45:41.361640 28330 tablet_bootstrap.cc:492] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: Bootstrap starting.
I20260806 11:45:41.362545 28018 tablet_service.cc:1467] Processing CreateTablet for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1 (DEFAULT_TABLE table=test-table [id=74fb7c01183a4b34aa3ff9d5157501d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:41.363422 28327 tablet_bootstrap.cc:654] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:41.363566 28018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:41.363826 28330 tablet_bootstrap.cc:654] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:41.364393 28327 log.cc:824] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:41.364660 28330 log.cc:824] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:41.366779 28330 tablet_bootstrap.cc:492] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: No bootstrap required, opened a new log
I20260806 11:45:41.366779 28327 tablet_bootstrap.cc:492] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7: No bootstrap required, opened a new log
I20260806 11:45:41.366992 28327 ts_tablet_manager.cc:1397] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 11:45:41.366992 28330 ts_tablet_manager.cc:1397] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 11:45:41.370184 28335 tablet_bootstrap.cc:492] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: Bootstrap starting.
I20260806 11:45:41.371541 28330 raft_consensus.cc:348] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } }
I20260806 11:45:41.371541 28327 raft_consensus.cc:348] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } }
I20260806 11:45:41.371838 28330 raft_consensus.cc:374] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:41.371845 28327 raft_consensus.cc:374] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:41.371897 28327 raft_consensus.cc:729] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 454d521eb36d475ca4e7ae4acc4938f7, State: Initialized, Role: FOLLOWER
I20260806 11:45:41.371907 28330 raft_consensus.cc:729] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a97059174374bcfa441c1499eeb0333, State: Initialized, Role: FOLLOWER
I20260806 11:45:41.372416 28335 tablet_bootstrap.cc:654] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:41.372435 28327 consensus_queue.cc:260] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } }
I20260806 11:45:41.372435 28330 consensus_queue.cc:260] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } }
I20260806 11:45:41.373394 27910 heartbeater.cc:503] Master 127.0.113.62:38359 was elected leader, sending a full tablet report...
I20260806 11:45:41.373517 28327 ts_tablet_manager.cc:1428] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 11:45:41.373517 28330 ts_tablet_manager.cc:1428] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 11:45:41.373911 28303 heartbeater.cc:503] Master 127.0.113.62:38359 was elected leader, sending a full tablet report...
I20260806 11:45:41.374431 28335 log.cc:824] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:41.376777 28335 tablet_bootstrap.cc:492] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: No bootstrap required, opened a new log
I20260806 11:45:41.377002 28335 ts_tablet_manager.cc:1397] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 11:45:41.380774 28335 raft_consensus.cc:348] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } }
I20260806 11:45:41.381022 28335 raft_consensus.cc:374] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:41.381103 28335 raft_consensus.cc:729] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58a0638d0794de7a6bbb05213c2751c, State: Initialized, Role: FOLLOWER
I20260806 11:45:41.381597 28335 consensus_queue.cc:260] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } }
I20260806 11:45:41.382370 28112 heartbeater.cc:503] Master 127.0.113.62:38359 was elected leader, sending a full tablet report...
I20260806 11:45:41.382639 28335 ts_tablet_manager.cc:1428] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 11:45:41.417878 28336 raft_consensus.cc:482] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:41.418141 28336 raft_consensus.cc:504] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } }
I20260806 11:45:41.419345 28336 leader_election.cc:288] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b58a0638d0794de7a6bbb05213c2751c (127.0.113.2:33563), 0a97059174374bcfa441c1499eeb0333 (127.0.113.3:42417)
I20260806 11:45:41.424017 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.419771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:59155 (local address 127.0.113.3:42417)
0806 11:45:41.419937 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:41.419942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:41.419961 (+ 19us) server_negotiation.cc:407] Connection header received
0806 11:45:41.420047 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:41.420050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:41.420104 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:41.420173 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:41.420525 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.421217 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.422731 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.423378 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.423503 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.423588 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:41.423777 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 11:45:41.423881 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 11:45:41.424032 27916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.419648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:42417 (local address 127.0.113.1:59155)
0806 11:45:41.419797 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:41.419814 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:41.419937 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:41.420213 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:41.420216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:41.420241 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:41.420401 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.420407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.421319 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.421325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.422599 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.422610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.423502 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.423505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.423638 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.423666 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 11:45:41.423747 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 11:45:41.423835 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"spinlock_wait_cycles":4736}
I20260806 11:45:41.424088 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.419767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:33089 (local address 127.0.113.2:33563)
0806 11:45:41.419895 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:41.419900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:41.420240 (+ 340us) server_negotiation.cc:407] Connection header received
0806 11:45:41.420357 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:41.420360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:41.420414 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:41.420492 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:41.420825 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.421446 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.422880 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.423491 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.423588 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.423667 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:41.423890 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 11:45:41.423969 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:45:41.424042 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.419648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:33563 (local address 127.0.113.1:33089)
0806 11:45:41.420113 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:41.420124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:41.420241 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:41.420550 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:41.420553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:41.420573 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:41.420692 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.420699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.421586 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.421590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.422749 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:41.422757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.423639 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.423642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:41.423743 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.423763 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 11:45:41.423827 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 11:45:41.423878 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":133,"spinlock_wait_cycles":384,"thread_start_us":103,"threads_started":1}
W20260806 11:45:41.424479 27915 tablet.cc:2369] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:45:41.424902 28238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900d4d7c40fe4aacb3fd3a5595e2b9b1" candidate_uuid: "454d521eb36d475ca4e7ae4acc4938f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a97059174374bcfa441c1499eeb0333" is_pre_election: true
I20260806 11:45:41.425132 28047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900d4d7c40fe4aacb3fd3a5595e2b9b1" candidate_uuid: "454d521eb36d475ca4e7ae4acc4938f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58a0638d0794de7a6bbb05213c2751c" is_pre_election: true
I20260806 11:45:41.425302 28238 raft_consensus.cc:2454] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 454d521eb36d475ca4e7ae4acc4938f7 in term 0.
I20260806 11:45:41.425436 28047 raft_consensus.cc:2454] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 454d521eb36d475ca4e7ae4acc4938f7 in term 0.
I20260806 11:45:41.425843 27757 leader_election.cc:302] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a97059174374bcfa441c1499eeb0333, 454d521eb36d475ca4e7ae4acc4938f7; no voters:
I20260806 11:45:41.426148 28336 raft_consensus.cc:2781] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:45:41.426255 28336 raft_consensus.cc:482] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:45:41.426326 28336 raft_consensus.cc:3037] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:41.427371 28336 raft_consensus.cc:504] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } }
I20260806 11:45:41.427790 28336 leader_election.cc:288] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [CANDIDATE]: Term 1 election: Requested vote from peers b58a0638d0794de7a6bbb05213c2751c (127.0.113.2:33563), 0a97059174374bcfa441c1499eeb0333 (127.0.113.3:42417)
I20260806 11:45:41.428170 28047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900d4d7c40fe4aacb3fd3a5595e2b9b1" candidate_uuid: "454d521eb36d475ca4e7ae4acc4938f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58a0638d0794de7a6bbb05213c2751c"
I20260806 11:45:41.428252 28238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900d4d7c40fe4aacb3fd3a5595e2b9b1" candidate_uuid: "454d521eb36d475ca4e7ae4acc4938f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a97059174374bcfa441c1499eeb0333"
I20260806 11:45:41.428337 28047 raft_consensus.cc:3037] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:41.428452 28238 raft_consensus.cc:3037] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:41.429431 28047 raft_consensus.cc:2454] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 454d521eb36d475ca4e7ae4acc4938f7 in term 1.
I20260806 11:45:41.429582 28238 raft_consensus.cc:2454] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 454d521eb36d475ca4e7ae4acc4938f7 in term 1.
I20260806 11:45:41.429813 27760 leader_election.cc:302] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 454d521eb36d475ca4e7ae4acc4938f7, b58a0638d0794de7a6bbb05213c2751c; no voters:
I20260806 11:45:41.430019 28336 raft_consensus.cc:2781] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:41.430236 28336 raft_consensus.cc:686] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [term 1 LEADER]: Becoming Leader. State: Replica: 454d521eb36d475ca4e7ae4acc4938f7, State: Running, Role: LEADER
I20260806 11:45:41.430565 28336 consensus_queue.cc:237] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } }
I20260806 11:45:41.433910 27648 catalog_manager.cc:5495] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 454d521eb36d475ca4e7ae4acc4938f7 (127.0.113.1). New cstate: current_term: 1 leader_uuid: "454d521eb36d475ca4e7ae4acc4938f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "454d521eb36d475ca4e7ae4acc4938f7" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 45861 } health_report { overall_health: HEALTHY } } }
I20260806 11:45:41.449512 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.445844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.0.113.1:45861)
0806 11:45:41.445987 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:41.445990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:41.446009 (+ 19us) server_negotiation.cc:407] Connection header received
0806 11:45:41.446146 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:41.446149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:41.446204 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:41.446292 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:45:41.446490 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.447003 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.447869 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:41.448115 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:41.448216 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:41.448578 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:45:41.449032 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:45:41.449164 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:41.449265 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 11:45:41.449340 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:45:41.456019 28047 raft_consensus.cc:1261] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [term 1 FOLLOWER]: Refusing update from remote peer 454d521eb36d475ca4e7ae4acc4938f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:41.456027 28238 raft_consensus.cc:1261] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [term 1 FOLLOWER]: Refusing update from remote peer 454d521eb36d475ca4e7ae4acc4938f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:41.456845 28336 consensus_queue.cc:1035] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a97059174374bcfa441c1499eeb0333" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:41.457104 28341 consensus_queue.cc:1035] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 454d521eb36d475ca4e7ae4acc4938f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b58a0638d0794de7a6bbb05213c2751c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:41.466485 28351 mvcc.cc:205] Tried to move back new op lower bound from 7315524572995596288 to 7315524572902268928. Current Snapshot: MvccSnapshot[applied={T|T < 7315524572995596288}]
I20260806 11:45:41.473519 27648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:41.473714 27648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:41.476208 27648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c47f345ec89749be975aee742e85a8d9: Sending DeleteTablet for 3 replicas of tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1
I20260806 11:45:41.477108 28018 tablet_service.cc:1514] Processing DeleteTablet for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:41 UTC) from {username='slave'} at 127.0.0.1:38218
I20260806 11:45:41.477348 28209 tablet_service.cc:1514] Processing DeleteTablet for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:41 UTC) from {username='slave'} at 127.0.0.1:46400
I20260806 11:45:41.477365 27819 tablet_service.cc:1514] Processing DeleteTablet for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:41 UTC) from {username='slave'} at 127.0.0.1:52968
I20260806 11:45:41.477880 28363 tablet_replica.cc:330] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: stopping tablet replica
I20260806 11:45:41.478245 28363 raft_consensus.cc:2229] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:45:41.478549 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 27730
I20260806 11:45:41.478612 28363 raft_consensus.cc:2258] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:41.484040 28363 ts_tablet_manager.cc:1905] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:41.485292 28362 tablet_replica.cc:330] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: stopping tablet replica
I20260806 11:45:41.485636 28362 raft_consensus.cc:2229] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:45:41.485977 28362 raft_consensus.cc:2258] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:41.487321 28362 ts_tablet_manager.cc:1905] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:41.487602 28363 ts_tablet_manager.cc:1918] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:41.487720 28363 log.cc:1196] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/900d4d7c40fe4aacb3fd3a5595e2b9b1
I20260806 11:45:41.488076 28363 ts_tablet_manager.cc:1939] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P 0a97059174374bcfa441c1499eeb0333: Deleting consensus metadata
I20260806 11:45:41.489583 27627 catalog_manager.cc:4841] TS 0a97059174374bcfa441c1499eeb0333 (127.0.113.3:42417): tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1 (table test-table [id=74fb7c01183a4b34aa3ff9d5157501d9]) successfully deleted
I20260806 11:45:41.490166 28362 ts_tablet_manager.cc:1918] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:41.490268 28362 log.cc:1196] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/900d4d7c40fe4aacb3fd3a5595e2b9b1
I20260806 11:45:41.490620 28362 ts_tablet_manager.cc:1939] T 900d4d7c40fe4aacb3fd3a5595e2b9b1 P b58a0638d0794de7a6bbb05213c2751c: Deleting consensus metadata
I20260806 11:45:41.491707 27629 catalog_manager.cc:4841] TS b58a0638d0794de7a6bbb05213c2751c (127.0.113.2:33563): tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1 (table test-table [id=74fb7c01183a4b34aa3ff9d5157501d9]) successfully deleted
W20260806 11:45:41.492449 27628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:45861 (error 108)
I20260806 11:45:41.493345 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 27917
I20260806 11:45:41.493595 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:41.493244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:45861 (local address 127.0.0.1:52980)
0806 11:45:41.493405 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:41.493491 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 11:45:41.500907 27628 catalog_manager.cc:4584] TS 454d521eb36d475ca4e7ae4acc4938f7 (127.0.113.1:45861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 900d4d7c40fe4aacb3fd3a5595e2b9b1: Network error: Client connection negotiation failed: client connection to 127.0.113.1:45861: connect: Connection refused (error 111)
I20260806 11:45:41.508852 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 28116
I20260806 11:45:41.523716 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 27604
2026-08-06T11:45:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1071 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:41 dist-test-slave-45dx krb5kdc[28369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:41 dist-test-slave-45dx krb5kdc[28369](info): set up 2 sockets
Aug 06 11:45:41 dist-test-slave-45dx krb5kdc[28369](info): commencing operation
krb5kdc: starting...
W20260806 11:45:42.775925 452 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:42 dist-test-slave-45dx krb5kdc[28369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:45:42.909729 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:40511
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:43.026115 28388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:43.026419 28388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:43.026492 28388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:43.030483 28388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:43.030567 28388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:43.030604 28388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:43.030663 28388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:43.030702 28388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:43.035849 28388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:40511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:43.037010 28388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:43.038060 28388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:43.046707 28399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:43.046736 28398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:43.046736 28402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:43.047140 28388 server_base.cc:1034] running on GCE node
I20260806 11:45:43.047786 28388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:43.049010 28388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:43.050204 28388 hybrid_clock.cc:648] HybridClock initialized: now 1786016743050166 us; error 49 us; skew 500 ppm
Aug 06 11:45:43 dist-test-slave-45dx krb5kdc[28369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:43.053077 28388 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:45:43.053987 28388 webserver.cc:460] Webserver started at http://127.0.113.62:37265/ using document root <none> and password file <none>
I20260806 11:45:43.054548 28388 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:43.054643 28388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:43.054877 28388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:43.057004 28388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "5c4c723819ed43ab8dfa894c2262edb8"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "0a83b0ed71eb88356114512aef0959eb"
server_key_iv: "0532b75dfced90a47c0928f35f839c23"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.057575 28388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "5c4c723819ed43ab8dfa894c2262edb8"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "0a83b0ed71eb88356114512aef0959eb"
server_key_iv: "0532b75dfced90a47c0928f35f839c23"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.061280 28388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 11:45:43.063825 28409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.065140 28388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:45:43.065289 28388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "5c4c723819ed43ab8dfa894c2262edb8"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "0a83b0ed71eb88356114512aef0959eb"
server_key_iv: "0532b75dfced90a47c0928f35f839c23"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.065407 28388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:43.078305 28388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:43.081646 28388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:43.081880 28388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:43.089862 28388 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:40511
I20260806 11:45:43.089881 28479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:40511 every 8 connection(s)
I20260806 11:45:43.090803 28388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:43.093535 28482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:43.097563 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 28388
I20260806 11:45:43.097743 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:43.098074 452 external_mini_cluster.cc:1428] Setting key 20a99ac75bc1a21f4b3e7b00c52373c1
I20260806 11:45:43.099273 28482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8: Bootstrap starting.
I20260806 11:45:43.101951 28482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:43.102783 28482 log.cc:824] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:43.104772 28482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8: No bootstrap required, opened a new log
Aug 06 11:45:43 dist-test-slave-45dx krb5kdc[28369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:45:43.108709 28482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4c723819ed43ab8dfa894c2262edb8" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40511 } }
I20260806 11:45:43.109006 28482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:43.109068 28482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4c723819ed43ab8dfa894c2262edb8, State: Initialized, Role: FOLLOWER
I20260806 11:45:43.109534 28482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4c723819ed43ab8dfa894c2262edb8" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40511 } }
I20260806 11:45:43.109699 28482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:43.109767 28482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:43.109858 28482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:43.110877 28482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4c723819ed43ab8dfa894c2262edb8" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40511 } }
I20260806 11:45:43.111212 28482 leader_election.cc:302] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c4c723819ed43ab8dfa894c2262edb8; no voters:
I20260806 11:45:43.111559 28482 leader_election.cc:288] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:43.111616 28487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:43.111793 28487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [term 1 LEADER]: Becoming Leader. State: Replica: 5c4c723819ed43ab8dfa894c2262edb8, State: Running, Role: LEADER
I20260806 11:45:43.112092 28487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4c723819ed43ab8dfa894c2262edb8" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40511 } }
I20260806 11:45:43.112948 28482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:43.113306 28487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4c723819ed43ab8dfa894c2262edb8. Latest consensus state: current_term: 1 leader_uuid: "5c4c723819ed43ab8dfa894c2262edb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4c723819ed43ab8dfa894c2262edb8" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40511 } } }
I20260806 11:45:43.113428 28487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:43.113541 28489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c4c723819ed43ab8dfa894c2262edb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4c723819ed43ab8dfa894c2262edb8" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 40511 } } }
I20260806 11:45:43.113653 28489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:43.113441 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.099547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.0.113.62:40511)
0806 11:45:43.099980 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:43.099998 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:43.100038 (+ 40us) server_negotiation.cc:407] Connection header received
0806 11:45:43.100923 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:43.100946 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:43.101334 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:43.101675 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:43.102467 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.103773 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.104893 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.105558 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.105717 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.108393 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:43.108400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:43.108415 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:43.108445 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:43.110394 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.110785 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.110793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.110801 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.110940 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.111250 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.111252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.111253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.111718 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:43.111945 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:43.112245 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 11:45:43.112591 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
W20260806 11:45:43.117513 28512 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:45:43.117625 28512 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:45:43.117697 28497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:43.118336 28497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:43.123620 28497 catalog_manager.cc:1347] Generated new cluster ID: 66894c73c06a4ac9849a29c93ebd232d
I20260806 11:45:43.123682 28497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:43.132083 28497 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:43.133347 28497 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:43.139946 28497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8: Generated new TSK 0
I20260806 11:45:43.140764 28497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
Entry for principal kudu/127.0.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab.
I20260806 11:45:43.169557 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40511
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:43.284615 28520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:43.284912 28520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:43.285001 28520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:43.289024 28520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:43.289081 28520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:43.289196 28520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:45:43.294480 28520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.1.keytab
--principal=kudu/127.0.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:43.295558 28520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:43.296627 28520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:43.306011 28525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:43.306006 28530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:43.306005 28533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:43.306360 28520 server_base.cc:1034] running on GCE node
I20260806 11:45:43.306828 28520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:43.307390 28520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:43.308544 28520 hybrid_clock.cc:648] HybridClock initialized: now 1786016743308514 us; error 41 us; skew 500 ppm
Aug 06 11:45:43 dist-test-slave-45dx krb5kdc[28369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:43.311179 28520 init.cc:377] Logged in from keytab as kudu/127.0.113.1@KRBTEST.COM (short username kudu)
I20260806 11:45:43.311949 28520 webserver.cc:460] Webserver started at http://127.0.113.1:44309/ using document root <none> and password file <none>
I20260806 11:45:43.312561 28520 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:43.312672 28520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:43.312848 28520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:43.315177 28520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "a1062b4c6a624b28b5e19949f6cf6c31"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "580ccfc247b4285476f125325e6990c5"
server_key_iv: "823625f5948e5170fae0aaf7bba18529"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.315778 28520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "a1062b4c6a624b28b5e19949f6cf6c31"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "580ccfc247b4285476f125325e6990c5"
server_key_iv: "823625f5948e5170fae0aaf7bba18529"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.319465 28520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 11:45:43.321825 28544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.323081 28520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:45:43.323225 28520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "a1062b4c6a624b28b5e19949f6cf6c31"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "580ccfc247b4285476f125325e6990c5"
server_key_iv: "823625f5948e5170fae0aaf7bba18529"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.323341 28520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:43.347920 28520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:43.351207 28520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:43.351433 28520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:43.352073 28520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:43.352958 28520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:43.353032 28520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.353106 28520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:43.353159 28520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.362708 28520 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:33925
I20260806 11:45:43.362739 28710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:33925 every 8 connection(s)
I20260806 11:45:43.363746 28520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:45:43.366197 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 28520
I20260806 11:45:43.366351 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:45:43.366786 452 external_mini_cluster.cc:1428] Setting key 7226e5e86d9e027e5cdb0f187443baef
Aug 06 11:45:43 dist-test-slave-45dx krb5kdc[28369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.1@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.113.2@KRBTEST.COM; defaulting to no policy
I20260806 11:45:43.378844 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.365716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:52555 (local address 127.0.113.62:40511)
0806 11:45:43.365878 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:43.365883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:43.366882 (+ 999us) server_negotiation.cc:407] Connection header received
0806 11:45:43.368110 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:43.368114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:43.368167 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:43.368253 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:43.369693 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.370570 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.371699 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.372072 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.372192 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.375025 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:43.375030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:43.375038 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:43.375064 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:43.376645 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.377198 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.377202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.377204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.377294 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.377669 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.377672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.377674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.377877 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:43.377992 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:43.378509 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 11:45:43.378646 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:45:43.379663 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.366136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40511 (local address 127.0.113.1:52555)
0806 11:45:43.366670 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:43.366739 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:43.367891 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:43.368468 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:43.368477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:43.368999 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:43.369527 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.369543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.370697 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.370700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.371566 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.371575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.372223 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.372226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.372441 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.373083 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:43.373099 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:43.374846 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:43.376774 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:43.376780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:43.376796 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:43.377084 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:43.377404 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:43.377407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:43.377409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:43.377560 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:43.377991 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:43.377998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:43.378255 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 11:45:43.378450 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 11:45:43.378805 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.2@KRBTEST.COM" created.
I20260806 11:45:43.381237 28711 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40511
I20260806 11:45:43.381522 28711 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:43.382157 28711 heartbeater.cc:511] Master 127.0.113.62:40511 requested a full tablet report, sending...
I20260806 11:45:43.384039 28431 ts_manager.cc:194] Registered new tserver with Master: a1062b4c6a624b28b5e19949f6cf6c31 (127.0.113.1:33925)
I20260806 11:45:43.385236 28431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.1@KRBTEST.COM'} at 127.0.113.1:52555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
Entry for principal kudu/127.0.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab.
I20260806 11:45:43.397660 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40511
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:43.510805 28720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:43.511055 28720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:43.511096 28720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:43.515194 28720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:43.515251 28720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:43.515304 28720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:45:43.520256 28720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.2.keytab
--principal=kudu/127.0.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:43.521224 28720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:43.522204 28720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:43.531450 28732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:43.531440 28729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:43.531577 28730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:43.531857 28720 server_base.cc:1034] running on GCE node
I20260806 11:45:43.532244 28720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:43.532943 28720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:43.534108 28720 hybrid_clock.cc:648] HybridClock initialized: now 1786016743534093 us; error 24 us; skew 500 ppm
Aug 06 11:45:43 dist-test-slave-45dx krb5kdc[28369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:43.537021 28720 init.cc:377] Logged in from keytab as kudu/127.0.113.2@KRBTEST.COM (short username kudu)
I20260806 11:45:43.537909 28720 webserver.cc:460] Webserver started at http://127.0.113.2:36485/ using document root <none> and password file <none>
I20260806 11:45:43.538466 28720 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:43.538528 28720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:43.538757 28720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:43.540799 28720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "7f6f6829e7544c7e8c005034aaa3b33c"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "83c247af3ee8a7bd4ced5416bbf38c25"
server_key_iv: "a43fd2f7ff524ee6efef6360cf4f8d06"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.541368 28720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "7f6f6829e7544c7e8c005034aaa3b33c"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "83c247af3ee8a7bd4ced5416bbf38c25"
server_key_iv: "a43fd2f7ff524ee6efef6360cf4f8d06"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.545053 28720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:45:43.547403 28747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.548718 28720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:45:43.548866 28720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "7f6f6829e7544c7e8c005034aaa3b33c"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "83c247af3ee8a7bd4ced5416bbf38c25"
server_key_iv: "a43fd2f7ff524ee6efef6360cf4f8d06"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.548979 28720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:43.589756 28720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:43.593443 28720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:43.593672 28720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:43.594343 28720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:43.595216 28720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:43.595296 28720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.595381 28720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:43.595436 28720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.604432 28720 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:35615
I20260806 11:45:43.604453 28917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:35615 every 8 connection(s)
I20260806 11:45:43.605376 28720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
Aug 06 11:45:43 dist-test-slave-45dx krb5kdc[28369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.2@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:45:43.615518 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 28720
I20260806 11:45:43.615659 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:45:43.615969 452 external_mini_cluster.cc:1428] Setting key a9e86d8514c28d9766c77e3c91d9a60f
I20260806 11:45:43.619544 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.607265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:40143 (local address 127.0.113.62:40511)
0806 11:45:43.607450 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:43.607455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:43.608320 (+ 865us) server_negotiation.cc:407] Connection header received
0806 11:45:43.609253 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:43.609256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:43.609310 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:43.609389 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:43.610552 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.611069 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.612174 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.612519 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.612601 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.615695 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:43.615699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:43.615702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:43.615729 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:43.617412 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.617889 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.617895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.617900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.617994 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.618331 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.618337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.618341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.618595 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:43.618701 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:43.619162 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 11:45:43.619296 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 11:45:43.620374 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.607660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40511 (local address 127.0.113.2:40143)
0806 11:45:43.608129 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:43.608175 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:43.609050 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:43.609559 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:43.609565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:43.609948 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:43.610417 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.610430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.611184 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.611186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.612057 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.612067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.612761 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.612764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.612911 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.613620 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:43.613646 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:43.615530 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:43.617509 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:43.617514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:43.617525 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:43.617781 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:43.618092 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:43.618094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:43.618095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:43.618230 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:43.618700 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:43.618705 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:43.618936 (+ 231us) client_negotiation.cc:769] Sending connection context
0806 11:45:43.619086 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 11:45:43.619320 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":117,"threads_started":1}
I20260806 11:45:43.621518 28922 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40511
I20260806 11:45:43.621757 28922 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:43.622287 28922 heartbeater.cc:511] Master 127.0.113.62:40511 requested a full tablet report, sending...
I20260806 11:45:43.623373 28431 ts_manager.cc:194] Registered new tserver with Master: 7f6f6829e7544c7e8c005034aaa3b33c (127.0.113.2:35615)
I20260806 11:45:43.623932 28431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.2@KRBTEST.COM'} at 127.0.113.2:40143
WARNING: no policy specified for kudu/127.0.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
Entry for principal kudu/127.0.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab.
I20260806 11:45:43.645491 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40511
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:43.759162 28932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:43.759450 28932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:43.759495 28932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:43.763780 28932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:43.763836 28932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:43.763896 28932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:45:43.769236 28932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/krb5kdc/kudu_127.0.113.3.keytab
--principal=kudu/127.0.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:43.770301 28932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:43.771299 28932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:43.780890 28937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:43.780936 28944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:43.781114 28932 server_base.cc:1034] running on GCE node
W20260806 11:45:43.780900 28938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:43.781733 28932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:43.782357 28932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:43.783555 28932 hybrid_clock.cc:648] HybridClock initialized: now 1786016743783538 us; error 27 us; skew 500 ppm
Aug 06 11:45:43 dist-test-slave-45dx krb5kdc[28369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:43.786252 28932 init.cc:377] Logged in from keytab as kudu/127.0.113.3@KRBTEST.COM (short username kudu)
I20260806 11:45:43.787112 28932 webserver.cc:460] Webserver started at http://127.0.113.3:44395/ using document root <none> and password file <none>
I20260806 11:45:43.787626 28932 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:43.787684 28932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:43.787842 28932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:43.789813 28932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "546ef211e0d549349298e75f11cbcb46"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "6e0f62526f373aacd85753f9528ffa14"
server_key_iv: "d35b5e6c225e030fa8845da584cc3611"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.790314 28932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "546ef211e0d549349298e75f11cbcb46"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "6e0f62526f373aacd85753f9528ffa14"
server_key_iv: "d35b5e6c225e030fa8845da584cc3611"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.793932 28932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260806 11:45:43.796103 28952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.797292 28932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:45:43.797392 28932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "546ef211e0d549349298e75f11cbcb46"
format_stamp: "Formatted at 2026-08-06 11:45:43 on dist-test-slave-45dx"
server_key: "6e0f62526f373aacd85753f9528ffa14"
server_key_iv: "d35b5e6c225e030fa8845da584cc3611"
server_key_version: "encryptionkey@0"
I20260806 11:45:43.797475 28932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:43.817204 28932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:43.820075 28932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:43.820231 28932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:43.820876 28932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:43.821697 28932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:43.821768 28932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.821870 28932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:43.821909 28932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:43.830993 28932 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:41817
I20260806 11:45:43.831019 29127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:41817 every 8 connection(s)
I20260806 11:45:43.831856 28932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:45:43.832520 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 28932
I20260806 11:45:43.832670 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:45:43.832983 452 external_mini_cluster.cc:1428] Setting key 44254878451d1086f27d79d378a5d03e
Aug 06 11:45:43 dist-test-slave-45dx krb5kdc[28369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.3@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:45:43.845027 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.833760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:35101 (local address 127.0.113.62:40511)
0806 11:45:43.833903 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:43.833907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:43.835021 (+ 1114us) server_negotiation.cc:407] Connection header received
0806 11:45:43.836053 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:43.836055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:43.836098 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:43.836166 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:43.837374 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.837895 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.838984 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.839194 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.839275 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.841899 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:43.841902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:43.841904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:43.841926 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:43.843156 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.843637 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.843640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.843641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.843713 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.844099 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.844101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.844102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.844257 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:43.844375 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:43.844821 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 11:45:43.844912 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 11:45:43.845760 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.834201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:40511 (local address 127.0.113.3:35101)
0806 11:45:43.834852 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:43.834913 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:43.835868 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:43.836389 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:43.836397 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:43.836766 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:43.837245 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.837258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.838002 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.838005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.838871 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.838880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.839325 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.839339 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.839478 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.840103 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:45:43.840120 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:43.841699 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:43.843262 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:43.843266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:43.843277 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:43.843548 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:43.843824 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:45:43.843828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:45:43.843831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:45:43.844016 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:45:43.844391 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:43.844396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:43.844603 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 11:45:43.844753 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 11:45:43.844983 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":150,"threads_started":1}
I20260806 11:45:43.846856 29128 heartbeater.cc:347] Connected to a master server at 127.0.113.62:40511
I20260806 11:45:43.847108 29128 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:43.847635 29128 heartbeater.cc:511] Master 127.0.113.62:40511 requested a full tablet report, sending...
I20260806 11:45:43.848706 28430 ts_manager.cc:194] Registered new tserver with Master: 546ef211e0d549349298e75f11cbcb46 (127.0.113.3:41817)
I20260806 11:45:43.849295 28430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.113.3@KRBTEST.COM'} at 127.0.113.3:35101
I20260806 11:45:43.859195 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:45:43.868861 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.861255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.0.113.62:40511)
0806 11:45:43.861439 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:43.861445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:43.861621 (+ 176us) server_negotiation.cc:407] Connection header received
0806 11:45:43.861803 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:43.861806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:43.861892 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:43.861982 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:43.862705 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.863219 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.864062 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.864284 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.864446 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.865847 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:43.865850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:43.865852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:43.865874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:43.867181 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.867583 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.867586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.867587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.867687 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:43.867938 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:43.867940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:43.867941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:43.868170 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:43.868270 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:43.868583 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 11:45:43.868718 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 11:45:43.871680 28430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:45:43.874611 28430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:45:43.889041 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.883895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:41817 (local address 127.0.0.1:57864)
0806 11:45:43.884186 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:43.884200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:43.884265 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:43.885187 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:43.885189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:43.885211 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:43.885310 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.885317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.886599 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.886603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.887619 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.887626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.888458 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.888462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.888567 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.888602 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 11:45:43.888749 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 11:45:43.888887 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
I20260806 11:45:43.889050 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.884124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.0.113.3:41817)
0806 11:45:43.884732 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:43.884743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:43.884765 (+ 22us) server_negotiation.cc:407] Connection header received
0806 11:45:43.884847 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:43.884852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:43.885027 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:43.885208 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:43.885437 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.886450 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.887747 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.888279 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.888453 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.888607 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:43.888783 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 11:45:43.888887 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":99,"threads_started":1}
I20260806 11:45:43.889194 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.882815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:35615 (local address 127.0.0.1:51930)
0806 11:45:43.883205 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:43.883259 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:43.883406 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:43.884252 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:43.884258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:43.884284 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:43.884460 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.884470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.885929 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.885932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.887346 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.887357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.888675 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.888679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.888874 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.888913 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 11:45:43.888974 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 11:45:43.889061 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":147,"thread_start_us":97,"threads_started":1}
I20260806 11:45:43.889295 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.883726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.0.113.1:33925)
0806 11:45:43.884557 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:43.884564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:43.884586 (+ 22us) server_negotiation.cc:407] Connection header received
0806 11:45:43.884657 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:43.884661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:43.884788 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:43.884942 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:43.885157 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.886031 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.887549 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.888355 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.888464 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.888606 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:43.889057 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 11:45:43.889152 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":81,"threads_started":1}
I20260806 11:45:43.889642 29148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.882980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.0.113.2:35615)
0806 11:45:43.883337 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:43.883350 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:43.883376 (+ 26us) server_negotiation.cc:407] Connection header received
0806 11:45:43.883623 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:43.883630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:43.883847 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:43.884028 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:43.884609 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.885782 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.887479 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.888355 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.888464 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.888606 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:43.889385 (+ 779us) server_negotiation.cc:299] Negotiation successful
0806 11:45:43.889485 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":152,"threads_started":1}
I20260806 11:45:43.889659 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:43.883565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:33925 (local address 127.0.0.1:43122)
0806 11:45:43.884140 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:43.884154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:43.884251 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:43.884921 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:43.884924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:43.884941 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:43.885027 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.885034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.886152 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.886155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.887362 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:43.887377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:43.888675 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:43.888679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:43.888833 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:43.888902 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 11:45:43.888975 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 11:45:43.889539 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":545,"thread_start_us":98,"threads_started":1}
I20260806 11:45:43.892407 29029 tablet_service.cc:1467] Processing CreateTablet for tablet 133c19eb65354227bd3705c8c1dd43b7 (DEFAULT_TABLE table=test-table [id=7531f13f521c42ef8034b352e7e3c4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:43.892405 28816 tablet_service.cc:1467] Processing CreateTablet for tablet 133c19eb65354227bd3705c8c1dd43b7 (DEFAULT_TABLE table=test-table [id=7531f13f521c42ef8034b352e7e3c4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:43.893602 28816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133c19eb65354227bd3705c8c1dd43b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:43.893602 29029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133c19eb65354227bd3705c8c1dd43b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:43.900476 28620 tablet_service.cc:1467] Processing CreateTablet for tablet 133c19eb65354227bd3705c8c1dd43b7 (DEFAULT_TABLE table=test-table [id=7531f13f521c42ef8034b352e7e3c4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:43.901081 29153 tablet_bootstrap.cc:492] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: Bootstrap starting.
I20260806 11:45:43.901566 28620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133c19eb65354227bd3705c8c1dd43b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:43.902277 29156 tablet_bootstrap.cc:492] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: Bootstrap starting.
I20260806 11:45:43.903678 29153 tablet_bootstrap.cc:654] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:43.904024 29156 tablet_bootstrap.cc:654] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:43.904562 29153 log.cc:824] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:43.905222 29156 log.cc:824] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:43.906745 29153 tablet_bootstrap.cc:492] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: No bootstrap required, opened a new log
I20260806 11:45:43.906896 29156 tablet_bootstrap.cc:492] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: No bootstrap required, opened a new log
I20260806 11:45:43.906962 29153 ts_tablet_manager.cc:1397] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 11:45:43.907095 29156 ts_tablet_manager.cc:1397] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 11:45:43.909104 29159 tablet_bootstrap.cc:492] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31: Bootstrap starting.
I20260806 11:45:43.910526 29153 raft_consensus.cc:348] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } }
I20260806 11:45:43.910517 29156 raft_consensus.cc:348] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } }
I20260806 11:45:43.910794 29153 raft_consensus.cc:374] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:43.910807 29156 raft_consensus.cc:374] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:43.910852 29153 raft_consensus.cc:729] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6f6829e7544c7e8c005034aaa3b33c, State: Initialized, Role: FOLLOWER
I20260806 11:45:43.910859 29156 raft_consensus.cc:729] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546ef211e0d549349298e75f11cbcb46, State: Initialized, Role: FOLLOWER
I20260806 11:45:43.911368 29159 tablet_bootstrap.cc:654] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:43.911295 29153 consensus_queue.cc:260] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } }
I20260806 11:45:43.911295 29156 consensus_queue.cc:260] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } }
I20260806 11:45:43.912138 29159 log.cc:824] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:43.912217 28922 heartbeater.cc:503] Master 127.0.113.62:40511 was elected leader, sending a full tablet report...
I20260806 11:45:43.912326 29153 ts_tablet_manager.cc:1428] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 11:45:43.912616 29128 heartbeater.cc:503] Master 127.0.113.62:40511 was elected leader, sending a full tablet report...
I20260806 11:45:43.912956 29156 ts_tablet_manager.cc:1428] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260806 11:45:43.914155 29159 tablet_bootstrap.cc:492] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31: No bootstrap required, opened a new log
I20260806 11:45:43.914340 29159 ts_tablet_manager.cc:1397] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:45:43.918342 29159 raft_consensus.cc:348] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } }
I20260806 11:45:43.918529 29159 raft_consensus.cc:374] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:43.918562 29159 raft_consensus.cc:729] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1062b4c6a624b28b5e19949f6cf6c31, State: Initialized, Role: FOLLOWER
I20260806 11:45:43.918883 29159 consensus_queue.cc:260] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } }
I20260806 11:45:43.919416 28711 heartbeater.cc:503] Master 127.0.113.62:40511 was elected leader, sending a full tablet report...
I20260806 11:45:43.919669 29159 ts_tablet_manager.cc:1428] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260806 11:45:44.083005 29129 tablet.cc:2369] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:45:44.106523 28924 tablet.cc:2369] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:45:44.115204 28712 tablet.cc:2369] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:45:44.139250 29162 raft_consensus.cc:482] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:44.139532 29162 raft_consensus.cc:504] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } }
I20260806 11:45:44.141211 29162 leader_election.cc:288] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6f6829e7544c7e8c005034aaa3b33c (127.0.113.2:35615), a1062b4c6a624b28b5e19949f6cf6c31 (127.0.113.1:33925)
I20260806 11:45:44.145915 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.141497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:35615 (local address 127.0.113.3:37877)
0806 11:45:44.141649 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:44.141665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:44.141795 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:44.142258 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:44.142263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:44.142286 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:44.142420 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:44.142430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.143329 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.143334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:44.144294 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:44.144304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.145462 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.145466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:44.145580 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.145614 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 11:45:44.145687 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 11:45:44.145767 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 11:45:44.146039 29148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.141609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:37877 (local address 127.0.113.2:35615)
0806 11:45:44.141796 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:44.141803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:44.141827 (+ 24us) server_negotiation.cc:407] Connection header received
0806 11:45:44.141994 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:44.141999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:44.142055 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:44.142133 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:44.142558 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.143210 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.144438 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.145305 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.145446 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.145544 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:44.145823 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 11:45:44.145913 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 11:45:44.146379 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.142095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:35991 (local address 127.0.113.1:33925)
0806 11:45:44.142243 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:44.142248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:44.142359 (+ 111us) server_negotiation.cc:407] Connection header received
0806 11:45:44.142468 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:44.142472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:44.142539 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:44.142647 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:44.142948 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.143831 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.145062 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.145575 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.145692 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.145751 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:44.146076 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 11:45:44.146183 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 11:45:44.146639 28845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133c19eb65354227bd3705c8c1dd43b7" candidate_uuid: "546ef211e0d549349298e75f11cbcb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" is_pre_election: true
I20260806 11:45:44.146934 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.141850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:33925 (local address 127.0.113.3:35991)
0806 11:45:44.142257 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:44.142270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:44.142371 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:44.142672 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:44.142675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:44.142697 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:44.142810 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:44.142816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.143987 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.143990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:44.144923 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:44.144932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.145721 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.145727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:44.145874 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.145913 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 11:45:44.146600 (+ 687us) client_negotiation.cc:240] Negotiation successful
0806 11:45:44.146707 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":119,"threads_started":1}
I20260806 11:45:44.147066 28845 raft_consensus.cc:2454] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546ef211e0d549349298e75f11cbcb46 in term 0.
I20260806 11:45:44.147583 28958 leader_election.cc:302] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546ef211e0d549349298e75f11cbcb46, 7f6f6829e7544c7e8c005034aaa3b33c; no voters:
I20260806 11:45:44.147759 28647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133c19eb65354227bd3705c8c1dd43b7" candidate_uuid: "546ef211e0d549349298e75f11cbcb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" is_pre_election: true
I20260806 11:45:44.147924 29162 raft_consensus.cc:2781] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:45:44.148072 29162 raft_consensus.cc:482] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:45:44.148083 28647 raft_consensus.cc:2454] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546ef211e0d549349298e75f11cbcb46 in term 0.
I20260806 11:45:44.148165 29162 raft_consensus.cc:3037] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:44.149857 29162 raft_consensus.cc:504] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } }
I20260806 11:45:44.150338 29162 leader_election.cc:288] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [CANDIDATE]: Term 1 election: Requested vote from peers 7f6f6829e7544c7e8c005034aaa3b33c (127.0.113.2:35615), a1062b4c6a624b28b5e19949f6cf6c31 (127.0.113.1:33925)
I20260806 11:45:44.150662 28845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133c19eb65354227bd3705c8c1dd43b7" candidate_uuid: "546ef211e0d549349298e75f11cbcb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6f6829e7544c7e8c005034aaa3b33c"
I20260806 11:45:44.150740 28647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133c19eb65354227bd3705c8c1dd43b7" candidate_uuid: "546ef211e0d549349298e75f11cbcb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1062b4c6a624b28b5e19949f6cf6c31"
I20260806 11:45:44.150800 28845 raft_consensus.cc:3037] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:44.150856 28647 raft_consensus.cc:3037] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:44.152144 28647 raft_consensus.cc:2454] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546ef211e0d549349298e75f11cbcb46 in term 1.
I20260806 11:45:44.152144 28845 raft_consensus.cc:2454] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546ef211e0d549349298e75f11cbcb46 in term 1.
I20260806 11:45:44.152469 28958 leader_election.cc:302] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546ef211e0d549349298e75f11cbcb46, 7f6f6829e7544c7e8c005034aaa3b33c; no voters:
I20260806 11:45:44.152661 29162 raft_consensus.cc:2781] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:44.152936 29162 raft_consensus.cc:686] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 1 LEADER]: Becoming Leader. State: Replica: 546ef211e0d549349298e75f11cbcb46, State: Running, Role: LEADER
I20260806 11:45:44.153537 29162 consensus_queue.cc:237] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } }
I20260806 11:45:44.157393 28431 catalog_manager.cc:5495] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 546ef211e0d549349298e75f11cbcb46 (127.0.113.3). New cstate: current_term: 1 leader_uuid: "546ef211e0d549349298e75f11cbcb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "546ef211e0d549349298e75f11cbcb46" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 41817 } health_report { overall_health: HEALTHY } } }
I20260806 11:45:44.181537 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.176872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.0.113.3:41817)
0806 11:45:44.177033 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:44.177039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:44.177061 (+ 22us) server_negotiation.cc:407] Connection header received
0806 11:45:44.177166 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:44.177173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:44.177258 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:44.177394 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:45:44.177651 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.178566 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.179426 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.179873 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.180013 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.180359 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:45:44.180991 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:45:44.181129 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:44.181209 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 11:45:44.181286 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 11:45:44.189358 28647 raft_consensus.cc:1261] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31 [term 1 FOLLOWER]: Refusing update from remote peer 546ef211e0d549349298e75f11cbcb46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:44.189358 28845 raft_consensus.cc:1261] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [term 1 FOLLOWER]: Refusing update from remote peer 546ef211e0d549349298e75f11cbcb46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:44.190150 29162 consensus_queue.cc:1035] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1062b4c6a624b28b5e19949f6cf6c31" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 33925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:44.190469 29168 consensus_queue.cc:1035] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6f6829e7544c7e8c005034aaa3b33c" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:44.200548 29176 mvcc.cc:205] Tried to move back new op lower bound from 7315524584189464576 to 7315524584055660544. Current Snapshot: MvccSnapshot[applied={T|T < 7315524584189464576}]
I20260806 11:45:44.212157 28431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:44.212414 28431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:44.216039 28431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c4c723819ed43ab8dfa894c2262edb8: Sending DeleteTablet for 3 replicas of tablet 133c19eb65354227bd3705c8c1dd43b7
I20260806 11:45:44.217171 28816 tablet_service.cc:1514] Processing DeleteTablet for tablet 133c19eb65354227bd3705c8c1dd43b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:44 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:51930
I20260806 11:45:44.217458 28620 tablet_service.cc:1514] Processing DeleteTablet for tablet 133c19eb65354227bd3705c8c1dd43b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:44 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:43122
I20260806 11:45:44.217973 29029 tablet_service.cc:1514] Processing DeleteTablet for tablet 133c19eb65354227bd3705c8c1dd43b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:44 UTC) from {username='kudu', principal='kudu/127.0.113.62@KRBTEST.COM'} at 127.0.0.1:57864
I20260806 11:45:44.218575 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 28520
I20260806 11:45:44.218823 29189 tablet_replica.cc:330] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: stopping tablet replica
I20260806 11:45:44.218634 29188 tablet_replica.cc:330] T 133c19eb65354227bd3705c8c1dd43b7 P a1062b4c6a624b28b5e19949f6cf6c31: stopping tablet replica
I20260806 11:45:44.219383 29189 raft_consensus.cc:2229] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:45:44.222056 29189 raft_consensus.cc:2258] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:44.222376 29187 tablet_replica.cc:330] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: stopping tablet replica
I20260806 11:45:44.222945 29187 raft_consensus.cc:2229] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:45:44.224612 29189 ts_tablet_manager.cc:1905] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:44.225011 29187 raft_consensus.cc:2258] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:44.227367 29187 ts_tablet_manager.cc:1905] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:44.231705 29187 ts_tablet_manager.cc:1918] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:44.233824 29189 ts_tablet_manager.cc:1918] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:44.234004 29189 log.cc:1196] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/133c19eb65354227bd3705c8c1dd43b7
I20260806 11:45:44.234421 29189 ts_tablet_manager.cc:1939] T 133c19eb65354227bd3705c8c1dd43b7 P 546ef211e0d549349298e75f11cbcb46: Deleting consensus metadata
I20260806 11:45:44.235841 28414 catalog_manager.cc:4841] TS 546ef211e0d549349298e75f11cbcb46 (127.0.113.3:41817): tablet 133c19eb65354227bd3705c8c1dd43b7 (table test-table [id=7531f13f521c42ef8034b352e7e3c4ea]) successfully deleted
I20260806 11:45:44.236169 29187 log.cc:1196] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/133c19eb65354227bd3705c8c1dd43b7
I20260806 11:45:44.236655 29187 ts_tablet_manager.cc:1939] T 133c19eb65354227bd3705c8c1dd43b7 P 7f6f6829e7544c7e8c005034aaa3b33c: Deleting consensus metadata
I20260806 11:45:44.237969 28414 catalog_manager.cc:4841] TS 7f6f6829e7544c7e8c005034aaa3b33c (127.0.113.2:35615): tablet 133c19eb65354227bd3705c8c1dd43b7 (table test-table [id=7531f13f521c42ef8034b352e7e3c4ea]) successfully deleted
W20260806 11:45:44.239797 28414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:33925 (error 108)
I20260806 11:45:44.241076 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.240737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:33925 (local address 127.0.0.1:43130)
0806 11:45:44.240907 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:44.240983 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:33925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 11:45:44.241266 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 28720
W20260806 11:45:44.241353 28414 catalog_manager.cc:4584] TS a1062b4c6a624b28b5e19949f6cf6c31 (127.0.113.1:33925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 133c19eb65354227bd3705c8c1dd43b7: Network error: Client connection negotiation failed: client connection to 127.0.113.1:33925: connect: Connection refused (error 111)
I20260806 11:45:44.252045 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 28932
I20260806 11:45:44.263234 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 28388
2026-08-06T11:45:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2745 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T11:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:44Z Disabled control of system clock
I20260806 11:45:44.310196 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:46205
--webserver_interface=127.0.113.62
--webserver_port=0
--builtin_ntp_servers=127.0.113.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:46205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 11:45:44.423161 29200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:44.423491 29200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:44.423563 29200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:44.427634 29200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:44.427731 29200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:44.427763 29200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:44.427832 29200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:44.427873 29200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:44.432780 29200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:46205
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:46205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:44.433971 29200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:44.435750 29200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:44.444751 29209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:44.444754 29210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:44.444963 29200 server_base.cc:1034] running on GCE node
W20260806 11:45:44.444772 29213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:44.445714 29200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:44.446800 29200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:44.447993 29200 hybrid_clock.cc:648] HybridClock initialized: now 1786016744447967 us; error 50 us; skew 500 ppm
I20260806 11:45:44.454892 29200 webserver.cc:460] Webserver started at http://127.0.113.62:38105/ using document root <none> and password file <none>
I20260806 11:45:44.455508 29200 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:44.455605 29200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:44.455847 29200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:44.458035 29200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f"
format_stamp: "Formatted at 2026-08-06 11:45:44 on dist-test-slave-45dx"
server_key: "c4bf0898fe97078895cae3e560c102df"
server_key_iv: "77247ab54e633663a37e2caa607f5e98"
server_key_version: "encryptionkey@0"
I20260806 11:45:44.458622 29200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f"
format_stamp: "Formatted at 2026-08-06 11:45:44 on dist-test-slave-45dx"
server_key: "c4bf0898fe97078895cae3e560c102df"
server_key_iv: "77247ab54e633663a37e2caa607f5e98"
server_key_version: "encryptionkey@0"
I20260806 11:45:44.462426 29200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:45:44.464965 29222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:44.466166 29200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:45:44.466315 29200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f"
format_stamp: "Formatted at 2026-08-06 11:45:44 on dist-test-slave-45dx"
server_key: "c4bf0898fe97078895cae3e560c102df"
server_key_iv: "77247ab54e633663a37e2caa607f5e98"
server_key_version: "encryptionkey@0"
I20260806 11:45:44.466439 29200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:44.481180 29200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:44.481937 29200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:44.482149 29200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:44.489773 29200 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:46205
I20260806 11:45:44.489797 29301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:46205 every 8 connection(s)
I20260806 11:45:44.490779 29200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:44.494225 29304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:44.496558 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 29200
I20260806 11:45:44.496701 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:44.497007 452 external_mini_cluster.cc:1428] Setting key ee9522b2d4bd2da2bfe0c9cf4aeb28f5
I20260806 11:45:44.500798 29304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f: Bootstrap starting.
I20260806 11:45:44.503230 29304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:44.504065 29304 log.cc:824] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:44.506321 29304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f: No bootstrap required, opened a new log
I20260806 11:45:44.506503 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.498557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.0.113.62:46205)
0806 11:45:44.499277 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:44.499293 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:44.499329 (+ 36us) server_negotiation.cc:407] Connection header received
0806 11:45:44.499981 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:44.499998 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:44.500481 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:44.500802 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:44.501223 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.502185 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.503091 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.503624 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.503799 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.504784 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:44.504790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:44.504809 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:44.505195 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:44.505253 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:44.505274 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:44.505392 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:44.505500 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 11:45:44.505762 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":165,"threads_started":1}
I20260806 11:45:44.510456 29304 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 46205 } }
I20260806 11:45:44.510689 29304 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:44.510788 29304 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2989ffc76a4477ea0eeb55daa1a5a8f, State: Initialized, Role: FOLLOWER
I20260806 11:45:44.511257 29304 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 46205 } }
I20260806 11:45:44.511421 29304 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:44.511497 29304 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:44.511623 29304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:44.512671 29304 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 46205 } }
I20260806 11:45:44.513103 29304 leader_election.cc:302] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2989ffc76a4477ea0eeb55daa1a5a8f; no voters:
I20260806 11:45:44.513468 29304 leader_election.cc:288] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:44.513586 29312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:44.513825 29312 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [term 1 LEADER]: Becoming Leader. State: Replica: c2989ffc76a4477ea0eeb55daa1a5a8f, State: Running, Role: LEADER
I20260806 11:45:44.514123 29312 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 46205 } }
I20260806 11:45:44.514930 29304 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:44.515434 29316 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2989ffc76a4477ea0eeb55daa1a5a8f. Latest consensus state: current_term: 1 leader_uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 46205 } } }
I20260806 11:45:44.515568 29316 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:44.515817 29313 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2989ffc76a4477ea0eeb55daa1a5a8f" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 46205 } } }
I20260806 11:45:44.515965 29313 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:44.515977 29320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:44.519253 29320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:44.525060 29320 catalog_manager.cc:1347] Generated new cluster ID: aa69dbaaab5442fe92bc8a4e64b3ec18
I20260806 11:45:44.525151 29320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:44.534217 29320 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:44.535538 29320 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:44.551474 29320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f: Generated new TSK 0
I20260806 11:45:44.552495 29320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:44.558243 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.1:0
--local_ip_for_outbound_sockets=127.0.113.1
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:46205
--builtin_ntp_servers=127.0.113.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 11:45:44.674769 29347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:44.675110 29347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:44.675150 29347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:44.679203 29347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:44.679258 29347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:44.679348 29347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.1
I20260806 11:45:44.684052 29347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.113.1
--webserver_port=0
--tserver_master_addrs=127.0.113.62:46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.1
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:44.685261 29347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:44.686679 29347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:44.696398 29352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:44.696414 29353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:44.696424 29356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:44.696647 29347 server_base.cc:1034] running on GCE node
I20260806 11:45:44.697136 29347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:44.697737 29347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:44.698967 29347 hybrid_clock.cc:648] HybridClock initialized: now 1786016744698952 us; error 24 us; skew 500 ppm
I20260806 11:45:44.700961 29347 webserver.cc:460] Webserver started at http://127.0.113.1:39803/ using document root <none> and password file <none>
I20260806 11:45:44.701558 29347 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:44.701658 29347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:44.701906 29347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:44.704015 29347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/data/instance:
uuid: "0f0148e60018489bb0d5efbcc513a012"
format_stamp: "Formatted at 2026-08-06 11:45:44 on dist-test-slave-45dx"
server_key: "84c909ebb21e2ce1eb1d4ec83f28d7d8"
server_key_iv: "c2d0a3bac28b1887748d9472bc49f524"
server_key_version: "encryptionkey@0"
I20260806 11:45:44.704628 29347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/wal/instance:
uuid: "0f0148e60018489bb0d5efbcc513a012"
format_stamp: "Formatted at 2026-08-06 11:45:44 on dist-test-slave-45dx"
server_key: "84c909ebb21e2ce1eb1d4ec83f28d7d8"
server_key_iv: "c2d0a3bac28b1887748d9472bc49f524"
server_key_version: "encryptionkey@0"
I20260806 11:45:44.708437 29347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 11:45:44.710913 29366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:44.712218 29347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:44.712404 29347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/wal
uuid: "0f0148e60018489bb0d5efbcc513a012"
format_stamp: "Formatted at 2026-08-06 11:45:44 on dist-test-slave-45dx"
server_key: "84c909ebb21e2ce1eb1d4ec83f28d7d8"
server_key_iv: "c2d0a3bac28b1887748d9472bc49f524"
server_key_version: "encryptionkey@0"
I20260806 11:45:44.712529 29347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:44.738070 29347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:44.738809 29347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:44.739029 29347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:44.739660 29347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:44.740710 29347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:44.740782 29347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:44.740852 29347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:44.740902 29347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:44.750151 29347 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.1:34301
I20260806 11:45:44.750240 29527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.1:34301 every 8 connection(s)
I20260806 11:45:44.751138 29347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/data/info.pb
I20260806 11:45:44.756428 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 29347
I20260806 11:45:44.756558 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-0/wal/instance
I20260806 11:45:44.756887 452 external_mini_cluster.cc:1428] Setting key aee323c1983406cbc13764e21502fdf2
I20260806 11:45:44.759003 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.2:0
--local_ip_for_outbound_sockets=127.0.113.2
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:46205
--builtin_ntp_servers=127.0.113.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 11:45:44.761368 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.753043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.1:42703 (local address 127.0.113.62:46205)
0806 11:45:44.753191 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:44.753195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:44.754196 (+ 1001us) server_negotiation.cc:407] Connection header received
0806 11:45:44.755626 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:44.755630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:44.755693 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:44.755803 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:44.757100 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.757737 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.759139 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.759530 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.759653 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.760828 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:44.760832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:44.760834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:44.760848 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:44.760894 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:44.760902 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:44.760980 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:44.761153 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 11:45:44.761226 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 11:45:44.762283 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.753371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:46205 (local address 127.0.113.1:42703)
0806 11:45:44.754030 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:44.754073 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:44.755389 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:44.755987 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:44.755995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:44.756125 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:44.756873 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:44.756894 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.757894 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.757898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:44.758999 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:44.759009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.759695 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.759699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:44.759876 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.760563 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:45:44.760565 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:44.760607 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:45:44.760673 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:44.760996 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:44.761007 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:44.761018 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 11:45:44.761141 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 11:45:44.761439 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260806 11:45:44.763564 29528 heartbeater.cc:347] Connected to a master server at 127.0.113.62:46205
I20260806 11:45:44.763829 29528 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:44.764561 29528 heartbeater.cc:511] Master 127.0.113.62:46205 requested a full tablet report, sending...
I20260806 11:45:44.766201 29247 ts_manager.cc:194] Registered new tserver with Master: 0f0148e60018489bb0d5efbcc513a012 (127.0.113.1:34301)
I20260806 11:45:44.767362 29247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.1:42703
W20260806 11:45:44.879532 29532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:44.879832 29532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:44.879914 29532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:44.884068 29532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:44.884126 29532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:44.884212 29532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.2
I20260806 11:45:44.889148 29532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.113.2
--webserver_port=0
--tserver_master_addrs=127.0.113.62:46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.2
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:44.890228 29532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:44.891695 29532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:44.901523 29542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:44.901574 29540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:44.901583 29544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:44.902027 29532 server_base.cc:1034] running on GCE node
I20260806 11:45:44.902452 29532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:44.903123 29532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:44.904358 29532 hybrid_clock.cc:648] HybridClock initialized: now 1786016744904340 us; error 36 us; skew 500 ppm
I20260806 11:45:44.906358 29532 webserver.cc:460] Webserver started at http://127.0.113.2:46825/ using document root <none> and password file <none>
I20260806 11:45:44.906981 29532 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:44.907045 29532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:44.907213 29532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:44.909368 29532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/data/instance:
uuid: "9d16d3d37a484c249f946f1ef31c5ed7"
format_stamp: "Formatted at 2026-08-06 11:45:44 on dist-test-slave-45dx"
server_key: "b4d72bd29bd1aef9e0e9aa3ff961a92a"
server_key_iv: "9531090c6b532ef81771c3151e530a77"
server_key_version: "encryptionkey@0"
I20260806 11:45:44.909915 29532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/wal/instance:
uuid: "9d16d3d37a484c249f946f1ef31c5ed7"
format_stamp: "Formatted at 2026-08-06 11:45:44 on dist-test-slave-45dx"
server_key: "b4d72bd29bd1aef9e0e9aa3ff961a92a"
server_key_iv: "9531090c6b532ef81771c3151e530a77"
server_key_version: "encryptionkey@0"
I20260806 11:45:44.913623 29532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:45:44.916196 29552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:44.917728 29532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:45:44.917865 29532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/wal
uuid: "9d16d3d37a484c249f946f1ef31c5ed7"
format_stamp: "Formatted at 2026-08-06 11:45:44 on dist-test-slave-45dx"
server_key: "b4d72bd29bd1aef9e0e9aa3ff961a92a"
server_key_iv: "9531090c6b532ef81771c3151e530a77"
server_key_version: "encryptionkey@0"
I20260806 11:45:44.917992 29532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:44.938211 29532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:44.938969 29532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:44.939183 29532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:44.939810 29532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:44.940850 29532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:44.940924 29532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:44.940999 29532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:44.941051 29532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:44.950253 29532 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.2:39033
I20260806 11:45:44.950264 29724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.2:39033 every 8 connection(s)
I20260806 11:45:44.951190 29532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/data/info.pb
I20260806 11:45:44.957486 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 29532
I20260806 11:45:44.957623 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/wal/instance
I20260806 11:45:44.957952 452 external_mini_cluster.cc:1428] Setting key 9efd01f8b1fb84d3cac38015d34b8300
I20260806 11:45:44.960268 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.113.3:0
--local_ip_for_outbound_sockets=127.0.113.3
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:46205
--builtin_ntp_servers=127.0.113.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 11:45:44.961167 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.953117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:52517 (local address 127.0.113.62:46205)
0806 11:45:44.953260 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:44.953264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:44.954321 (+ 1057us) server_negotiation.cc:407] Connection header received
0806 11:45:44.955776 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:44.955780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:44.955844 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:44.955959 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:44.956900 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.957612 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.959002 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.959329 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.959445 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.960709 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:44.960712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:44.960714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:44.960725 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:44.960746 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:44.960754 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:44.960814 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:44.961004 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 11:45:44.961055 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 11:45:44.962440 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:44.953480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:46205 (local address 127.0.113.2:52517)
0806 11:45:44.954153 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:44.954194 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:44.955546 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:44.956137 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:44.956144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:44.956227 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:44.956739 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:44.956752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.957796 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.957800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:44.958876 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:44.958886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:44.959518 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:44.959523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:44.959717 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:44.960402 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:45:44.960406 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:44.960466 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:45:44.960553 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:44.960842 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:44.960853 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:44.960862 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 11:45:44.961001 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 11:45:44.961327 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260806 11:45:44.963783 29725 heartbeater.cc:347] Connected to a master server at 127.0.113.62:46205
I20260806 11:45:44.964047 29725 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:44.964699 29725 heartbeater.cc:511] Master 127.0.113.62:46205 requested a full tablet report, sending...
I20260806 11:45:44.965974 29247 ts_manager.cc:194] Registered new tserver with Master: 9d16d3d37a484c249f946f1ef31c5ed7 (127.0.113.2:39033)
I20260806 11:45:44.966884 29247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.2:52517
W20260806 11:45:45.080524 29729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:45.080785 29729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:45.080832 29729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:45.084797 29729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:45.084856 29729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:45.084910 29729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.113.3
I20260806 11:45:45.089537 29729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.113.3
--webserver_port=0
--tserver_master_addrs=127.0.113.62:46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.113.3
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:45.090600 29729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:45.091917 29729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:45.101760 29740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:45.101756 29736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:45.101982 29737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:45.102072 29729 server_base.cc:1034] running on GCE node
I20260806 11:45:45.102470 29729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:45.103155 29729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:45.104386 29729 hybrid_clock.cc:648] HybridClock initialized: now 1786016745104371 us; error 28 us; skew 500 ppm
I20260806 11:45:45.106341 29729 webserver.cc:460] Webserver started at http://127.0.113.3:43685/ using document root <none> and password file <none>
I20260806 11:45:45.106976 29729 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:45.107077 29729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:45.107317 29729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:45.109441 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/data/instance:
uuid: "e9107669e6d34f0c89770d973b30562e"
format_stamp: "Formatted at 2026-08-06 11:45:45 on dist-test-slave-45dx"
server_key: "686e3b3190d86e4d52b7d70fc62bbaba"
server_key_iv: "56b0f6a7fb7726b743133b4a2bca6df3"
server_key_version: "encryptionkey@0"
I20260806 11:45:45.110033 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/wal/instance:
uuid: "e9107669e6d34f0c89770d973b30562e"
format_stamp: "Formatted at 2026-08-06 11:45:45 on dist-test-slave-45dx"
server_key: "686e3b3190d86e4d52b7d70fc62bbaba"
server_key_iv: "56b0f6a7fb7726b743133b4a2bca6df3"
server_key_version: "encryptionkey@0"
I20260806 11:45:45.113695 29729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:45:45.116428 29749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:45.117786 29729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:45:45.117888 29729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/wal
uuid: "e9107669e6d34f0c89770d973b30562e"
format_stamp: "Formatted at 2026-08-06 11:45:45 on dist-test-slave-45dx"
server_key: "686e3b3190d86e4d52b7d70fc62bbaba"
server_key_iv: "56b0f6a7fb7726b743133b4a2bca6df3"
server_key_version: "encryptionkey@0"
I20260806 11:45:45.117975 29729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:45.132212 29729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:45.132928 29729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:45.133123 29729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:45.133692 29729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 11:45:45.134603 29729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 11:45:45.134650 29729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:45.134688 29729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 11:45:45.134729 29729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:45.143985 29729 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.3:34621
I20260806 11:45:45.144018 29923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.3:34621 every 8 connection(s)
I20260806 11:45:45.144928 29729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/data/info.pb
I20260806 11:45:45.148571 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 29729
I20260806 11:45:45.148697 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/wal/instance
I20260806 11:45:45.149008 452 external_mini_cluster.cc:1428] Setting key 4244111bbaf24467789dfd25ec019090
I20260806 11:45:45.154376 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.146717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.3:57425 (local address 127.0.113.62:46205)
0806 11:45:45.146921 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:45.146925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:45.147954 (+ 1029us) server_negotiation.cc:407] Connection header received
0806 11:45:45.149546 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:45.149549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:45.149608 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:45.149698 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:45.150715 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.151381 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.152587 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.152811 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.152909 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.153980 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:45.153983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:45.153985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:45.153997 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:45.154016 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:45.154024 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:45.154082 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:45.154218 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 11:45:45.154266 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 11:45:45.155252 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.147141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:46205 (local address 127.0.113.3:57425)
0806 11:45:45.147791 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:45.147837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:45.149335 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:45.149921 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:45.149928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:45.150011 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:45:45.150570 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.150586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.151498 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.151501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.152472 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.152483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.152977 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.152980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.153122 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.153733 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 11:45:45.153735 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:45:45.153773 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 11:45:45.153841 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:45:45.154099 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:45:45.154106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:45:45.154109 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 11:45:45.154202 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 11:45:45.154412 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260806 11:45:45.156417 29925 heartbeater.cc:347] Connected to a master server at 127.0.113.62:46205
I20260806 11:45:45.156641 29925 heartbeater.cc:464] Registering TS with master...
I20260806 11:45:45.157136 29925 heartbeater.cc:511] Master 127.0.113.62:46205 requested a full tablet report, sending...
I20260806 11:45:45.158129 29247 ts_manager.cc:194] Registered new tserver with Master: e9107669e6d34f0c89770d973b30562e (127.0.113.3:34621)
I20260806 11:45:45.158670 29247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.113.3:57425
I20260806 11:45:45.163112 452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 11:45:45.168718 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.165097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.0.113.62:46205)
0806 11:45:45.165219 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:45.165222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:45.165448 (+ 226us) server_negotiation.cc:407] Connection header received
0806 11:45:45.165584 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:45.165586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:45.165623 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:45.165688 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:45.165968 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.166447 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.167281 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.167478 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.167552 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.168360 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:45.168364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:45.168367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:45.168380 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:45.168397 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:45.168406 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:45.168463 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:45.168573 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 11:45:45.168619 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:45:45.171491 29247 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 11:45:45.174633 29247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 11:45:45.189494 29946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.183195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:34301 (local address 127.0.0.1:41716)
0806 11:45:45.183732 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:45.183785 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:45.183930 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:45.184799 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:45.184806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:45.184850 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:45.185049 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.185056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.186579 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.186582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.187832 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.187841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.188957 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.188961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.189058 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.189085 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 11:45:45.189144 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 11:45:45.189294 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":52,"thread_start_us":97,"threads_started":1}
I20260806 11:45:45.189498 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.184015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.0.113.3:34621)
0806 11:45:45.184401 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:45.184414 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:45.184439 (+ 25us) server_negotiation.cc:407] Connection header received
0806 11:45:45.184525 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:45.184534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:45.184728 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:45.184944 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:45.185337 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.186460 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.188048 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.188764 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.188904 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.189109 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:45.189206 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 11:45:45.189321 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":126,"threads_started":1}
I20260806 11:45:45.189492 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.183438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:34621 (local address 127.0.0.1:36978)
0806 11:45:45.183874 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:45.183891 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:45.183993 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:45.184916 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:45.184918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:45.184932 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:45.185194 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.185200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.186628 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.186630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.187902 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.187913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.188903 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.188907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.189012 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.189059 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 11:45:45.189178 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 11:45:45.189313 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":138,"thread_start_us":120,"threads_started":1}
I20260806 11:45:45.190201 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.183930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.2:39033 (local address 127.0.0.1:39674)
0806 11:45:45.185357 (+ 1427us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:45.185380 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:45.185472 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:45.185895 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:45.185898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:45.185916 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:45.186014 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.186023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.187207 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.187210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.188283 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.188293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.189570 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.189794 (+ 224us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.189896 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.189919 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 11:45:45.190004 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 11:45:45.190074 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":108,"threads_started":1}
I20260806 11:45:45.190312 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.184402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.0.113.2:39033)
0806 11:45:45.184789 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:45.184801 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:45.185562 (+ 761us) server_negotiation.cc:407] Connection header received
0806 11:45:45.185628 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:45.185632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:45.185758 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:45.185905 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:45.186134 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.187080 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.188455 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.189210 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.189308 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.189497 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:45.190054 (+ 557us) server_negotiation.cc:299] Negotiation successful
0806 11:45:45.190161 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":134,"threads_started":1}
I20260806 11:45:45.190390 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.183420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.0.113.1:34301)
0806 11:45:45.184259 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:45.184272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:45.184330 (+ 58us) server_negotiation.cc:407] Connection header received
0806 11:45:45.184414 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:45.184418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:45.184607 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:45.185044 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:45.185197 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.186454 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.187982 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.188797 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.189892 (+ 1095us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.190079 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:45.190128 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 11:45:45.190218 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":133,"threads_started":1}
I20260806 11:45:45.192904 29825 tablet_service.cc:1467] Processing CreateTablet for tablet eed68a5f456d46b3b42c310d8ce93e0f (DEFAULT_TABLE table=test-table [id=fe4e860808c242959d8cec674e574218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:45.193094 29432 tablet_service.cc:1467] Processing CreateTablet for tablet eed68a5f456d46b3b42c310d8ce93e0f (DEFAULT_TABLE table=test-table [id=fe4e860808c242959d8cec674e574218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:45.193969 29825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed68a5f456d46b3b42c310d8ce93e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:45.193632 29626 tablet_service.cc:1467] Processing CreateTablet for tablet eed68a5f456d46b3b42c310d8ce93e0f (DEFAULT_TABLE table=test-table [id=fe4e860808c242959d8cec674e574218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 11:45:45.194114 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed68a5f456d46b3b42c310d8ce93e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:45.194518 29626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed68a5f456d46b3b42c310d8ce93e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:45.200932 29953 tablet_bootstrap.cc:492] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: Bootstrap starting.
I20260806 11:45:45.202028 29954 tablet_bootstrap.cc:492] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012: Bootstrap starting.
I20260806 11:45:45.203169 29953 tablet_bootstrap.cc:654] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:45.203687 29954 tablet_bootstrap.cc:654] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:45.203969 29953 log.cc:824] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:45.204509 29954 log.cc:824] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:45.204658 29955 tablet_bootstrap.cc:492] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: Bootstrap starting.
I20260806 11:45:45.206259 29953 tablet_bootstrap.cc:492] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: No bootstrap required, opened a new log
I20260806 11:45:45.206483 29953 ts_tablet_manager.cc:1397] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260806 11:45:45.206483 29954 tablet_bootstrap.cc:492] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012: No bootstrap required, opened a new log
I20260806 11:45:45.206621 29955 tablet_bootstrap.cc:654] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:45.206684 29954 ts_tablet_manager.cc:1397] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 11:45:45.207479 29955 log.cc:824] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:45.209470 29955 tablet_bootstrap.cc:492] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: No bootstrap required, opened a new log
I20260806 11:45:45.209664 29955 ts_tablet_manager.cc:1397] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 11:45:45.210194 29953 raft_consensus.cc:348] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } }
I20260806 11:45:45.210444 29953 raft_consensus.cc:374] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:45.210506 29953 raft_consensus.cc:729] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9107669e6d34f0c89770d973b30562e, State: Initialized, Role: FOLLOWER
I20260806 11:45:45.210986 29953 consensus_queue.cc:260] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } }
I20260806 11:45:45.210882 29954 raft_consensus.cc:348] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } }
I20260806 11:45:45.211145 29954 raft_consensus.cc:374] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:45.211200 29954 raft_consensus.cc:729] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0148e60018489bb0d5efbcc513a012, State: Initialized, Role: FOLLOWER
I20260806 11:45:45.211649 29925 heartbeater.cc:503] Master 127.0.113.62:46205 was elected leader, sending a full tablet report...
I20260806 11:45:45.211652 29954 consensus_queue.cc:260] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } }
I20260806 11:45:45.212019 29953 ts_tablet_manager.cc:1428] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260806 11:45:45.212603 29954 ts_tablet_manager.cc:1428] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 11:45:45.212927 29528 heartbeater.cc:503] Master 127.0.113.62:46205 was elected leader, sending a full tablet report...
I20260806 11:45:45.214818 29955 raft_consensus.cc:348] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } }
I20260806 11:45:45.215082 29955 raft_consensus.cc:374] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:45.215134 29955 raft_consensus.cc:729] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d16d3d37a484c249f946f1ef31c5ed7, State: Initialized, Role: FOLLOWER
I20260806 11:45:45.215538 29955 consensus_queue.cc:260] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } }
I20260806 11:45:45.216233 29725 heartbeater.cc:503] Master 127.0.113.62:46205 was elected leader, sending a full tablet report...
I20260806 11:45:45.216586 29955 ts_tablet_manager.cc:1428] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260806 11:45:45.252576 29530 tablet.cc:2369] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:45:45.395983 29927 tablet.cc:2369] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 11:45:45.452512 29726 tablet.cc:2369] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 11:45:45.511725 29964 raft_consensus.cc:482] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 11:45:45.511993 29964 raft_consensus.cc:504] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } }
I20260806 11:45:45.513170 29964 leader_election.cc:288] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f0148e60018489bb0d5efbcc513a012 (127.0.113.1:34301), e9107669e6d34f0c89770d973b30562e (127.0.113.3:34621)
I20260806 11:45:45.517560 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.513590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:42675 (local address 127.0.113.1:34301)
0806 11:45:45.513801 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:45.513807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:45.513912 (+ 105us) server_negotiation.cc:407] Connection header received
0806 11:45:45.514125 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:45.514128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:45.514179 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:45.514267 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:45.514569 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.515206 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.516628 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.517017 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.517155 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.517219 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:45.517369 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 11:45:45.517451 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260806 11:45:45.517583 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.513468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:34301 (local address 127.0.113.2:42675)
0806 11:45:45.513797 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:45.513811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:45.513989 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:45.514279 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:45.514281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:45.514297 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:45.514450 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.514456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.515313 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.515318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.516508 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.516518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.517144 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.517148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.517246 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.517273 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 11:45:45.517350 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 11:45:45.517421 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":90,"spinlock_wait_cycles":8320,"thread_start_us":85,"threads_started":1}
I20260806 11:45:45.517717 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.513590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.113.2:60773 (local address 127.0.113.3:34621)
0806 11:45:45.513732 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:45.513737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:45.513901 (+ 164us) server_negotiation.cc:407] Connection header received
0806 11:45:45.514126 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:45.514131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:45.514178 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:45.514260 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 11:45:45.514597 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.515135 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.516550 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.517118 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.517267 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.517326 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:45.517508 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 11:45:45.517573 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 11:45:45.517954 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.513468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.3:34621 (local address 127.0.113.2:60773)
0806 11:45:45.513781 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:45.513804 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 11:45:45.513988 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:45:45.514284 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:45:45.514287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:45:45.514309 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 11:45:45.514476 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.514481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.515245 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.515250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.516401 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:45:45.516438 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.517270 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.517274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:45:45.517383 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.517407 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 11:45:45.517480 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 11:45:45.517552 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":22,"spinlock_wait_cycles":49408,"thread_start_us":93,"threads_started":1}
I20260806 11:45:45.518312 29465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed68a5f456d46b3b42c310d8ce93e0f" candidate_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0148e60018489bb0d5efbcc513a012" is_pre_election: true
I20260806 11:45:45.518620 29465 raft_consensus.cc:2454] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d16d3d37a484c249f946f1ef31c5ed7 in term 0.
I20260806 11:45:45.518616 29859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed68a5f456d46b3b42c310d8ce93e0f" candidate_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9107669e6d34f0c89770d973b30562e" is_pre_election: true
I20260806 11:45:45.518885 29859 raft_consensus.cc:2454] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d16d3d37a484c249f946f1ef31c5ed7 in term 0.
I20260806 11:45:45.519130 29559 leader_election.cc:302] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0148e60018489bb0d5efbcc513a012, 9d16d3d37a484c249f946f1ef31c5ed7; no voters:
I20260806 11:45:45.519407 29964 raft_consensus.cc:2781] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 11:45:45.519510 29964 raft_consensus.cc:482] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 11:45:45.519574 29964 raft_consensus.cc:3037] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:45.520702 29964 raft_consensus.cc:504] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } }
I20260806 11:45:45.521066 29964 leader_election.cc:288] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [CANDIDATE]: Term 1 election: Requested vote from peers 0f0148e60018489bb0d5efbcc513a012 (127.0.113.1:34301), e9107669e6d34f0c89770d973b30562e (127.0.113.3:34621)
I20260806 11:45:45.521497 29859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed68a5f456d46b3b42c310d8ce93e0f" candidate_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9107669e6d34f0c89770d973b30562e"
I20260806 11:45:45.521481 29465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed68a5f456d46b3b42c310d8ce93e0f" candidate_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0148e60018489bb0d5efbcc513a012"
I20260806 11:45:45.521618 29465 raft_consensus.cc:3037] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:45.521618 29859 raft_consensus.cc:3037] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:45.522886 29859 raft_consensus.cc:2454] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d16d3d37a484c249f946f1ef31c5ed7 in term 1.
I20260806 11:45:45.522893 29465 raft_consensus.cc:2454] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d16d3d37a484c249f946f1ef31c5ed7 in term 1.
I20260806 11:45:45.523218 29561 leader_election.cc:302] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f0148e60018489bb0d5efbcc513a012, 9d16d3d37a484c249f946f1ef31c5ed7, e9107669e6d34f0c89770d973b30562e; no voters:
I20260806 11:45:45.523382 29964 raft_consensus.cc:2781] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:45.523599 29964 raft_consensus.cc:686] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 1 LEADER]: Becoming Leader. State: Replica: 9d16d3d37a484c249f946f1ef31c5ed7, State: Running, Role: LEADER
I20260806 11:45:45.523931 29964 consensus_queue.cc:237] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } }
I20260806 11:45:45.527197 29243 catalog_manager.cc:5495] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d16d3d37a484c249f946f1ef31c5ed7 (127.0.113.2). New cstate: current_term: 1 leader_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d16d3d37a484c249f946f1ef31c5ed7" member_type: VOTER last_known_addr { host: "127.0.113.2" port: 39033 } health_report { overall_health: HEALTHY } } }
I20260806 11:45:45.553813 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.550305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.0.113.2:39033)
0806 11:45:45.550472 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:45.550475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:45.550489 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:45:45.550567 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:45.550569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:45.550624 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:45.550717 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:45:45.550964 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.551465 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.552419 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:45.552625 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:45.552705 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:45.552985 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:45:45.553392 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:45:45.553479 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:45.553576 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 11:45:45.553622 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 11:45:45.559471 29465 raft_consensus.cc:1261] T eed68a5f456d46b3b42c310d8ce93e0f P 0f0148e60018489bb0d5efbcc513a012 [term 1 FOLLOWER]: Refusing update from remote peer 9d16d3d37a484c249f946f1ef31c5ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:45.559471 29859 raft_consensus.cc:1261] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [term 1 FOLLOWER]: Refusing update from remote peer 9d16d3d37a484c249f946f1ef31c5ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:45:45.560237 29969 consensus_queue.cc:1035] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9107669e6d34f0c89770d973b30562e" member_type: VOTER last_known_addr { host: "127.0.113.3" port: 34621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:45.560456 29964 consensus_queue.cc:1035] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0148e60018489bb0d5efbcc513a012" member_type: VOTER last_known_addr { host: "127.0.113.1" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:45:45.577471 29247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:45.577618 29247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 11:45:45.580224 29247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2989ffc76a4477ea0eeb55daa1a5a8f: Sending DeleteTablet for 3 replicas of tablet eed68a5f456d46b3b42c310d8ce93e0f
I20260806 11:45:45.581059 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet eed68a5f456d46b3b42c310d8ce93e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:45 UTC) from {username='slave'} at 127.0.0.1:41716
I20260806 11:45:45.581101 29626 tablet_service.cc:1514] Processing DeleteTablet for tablet eed68a5f456d46b3b42c310d8ce93e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:45 UTC) from {username='slave'} at 127.0.0.1:39674
I20260806 11:45:45.581101 29825 tablet_service.cc:1514] Processing DeleteTablet for tablet eed68a5f456d46b3b42c310d8ce93e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 11:45:45 UTC) from {username='slave'} at 127.0.0.1:36978
I20260806 11:45:45.581617 29991 tablet_replica.cc:330] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: stopping tablet replica
I20260806 11:45:45.581817 29988 tablet_replica.cc:330] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: stopping tablet replica
I20260806 11:45:45.582121 29988 raft_consensus.cc:2229] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 11:45:45.582118 29991 raft_consensus.cc:2229] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 11:45:45.582314 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 29347
I20260806 11:45:45.582535 29988 raft_consensus.cc:2258] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:45.591415 29988 ts_tablet_manager.cc:1905] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:45.591763 29991 raft_consensus.cc:2258] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 11:45:45.593534 29991 ts_tablet_manager.cc:1905] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 11:45:45.595438 29988 ts_tablet_manager.cc:1918] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:45.595553 29988 log.cc:1196] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-1/wal/wals/eed68a5f456d46b3b42c310d8ce93e0f
I20260806 11:45:45.595880 29988 ts_tablet_manager.cc:1939] T eed68a5f456d46b3b42c310d8ce93e0f P 9d16d3d37a484c249f946f1ef31c5ed7: Deleting consensus metadata
I20260806 11:45:45.597220 29223 catalog_manager.cc:4841] TS 9d16d3d37a484c249f946f1ef31c5ed7 (127.0.113.2:39033): tablet eed68a5f456d46b3b42c310d8ce93e0f (table test-table [id=fe4e860808c242959d8cec674e574218]) successfully deleted
I20260806 11:45:45.597456 29991 ts_tablet_manager.cc:1918] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 11:45:45.597572 29991 log.cc:1196] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: Deleting WAL directory at /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786016488822243-452-0/minicluster-data/ts-2/wal/wals/eed68a5f456d46b3b42c310d8ce93e0f
I20260806 11:45:45.597919 29991 ts_tablet_manager.cc:1939] T eed68a5f456d46b3b42c310d8ce93e0f P e9107669e6d34f0c89770d973b30562e: Deleting consensus metadata
W20260806 11:45:45.598985 29227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.113.1:34301 (error 108)
I20260806 11:45:45.599263 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 29532
I20260806 11:45:45.608366 29228 catalog_manager.cc:4841] TS e9107669e6d34f0c89770d973b30562e (127.0.113.3:34621): tablet eed68a5f456d46b3b42c310d8ce93e0f (table test-table [id=fe4e860808c242959d8cec674e574218]) successfully deleted
I20260806 11:45:45.614192 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:45.613800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.1:34301 (local address 127.0.0.1:41724)
0806 11:45:45.613978 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 11:45:45.614082 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.1:34301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 11:45:45.614405 29227 catalog_manager.cc:4584] TS 0f0148e60018489bb0d5efbcc513a012 (127.0.113.1:34301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eed68a5f456d46b3b42c310d8ce93e0f: Network error: Client connection negotiation failed: client connection to 127.0.113.1:34301: connect: Connection refused (error 111)
I20260806 11:45:45.628407 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 29729
I20260806 11:45:45.641034 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 29200
2026-08-06T11:45:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1365 ms)
[----------] 6 tests from EncryptionPolicyTest (11544 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:45 dist-test-slave-45dx krb5kdc[29997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:45 dist-test-slave-45dx krb5kdc[29997](info): set up 2 sockets
Aug 06 11:45:45 dist-test-slave-45dx krb5kdc[29997](info): commencing operation
krb5kdc: starting...
W20260806 11:45:46.854748 452 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:46 dist-test-slave-45dx krb5kdc[29997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:45:46.981158 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:47.094025 30019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:47.094274 30019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:47.094318 30019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:47.098171 30019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:47.098225 30019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:47.098245 30019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:47.098261 30019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:47.098318 30019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:47.103312 30019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:47.104290 30019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:47.105319 30019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:47.113482 30032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:47.113476 30030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:47.113656 30019 server_base.cc:1034] running on GCE node
W20260806 11:45:47.113476 30034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:47.114372 30019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:47.115465 30019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:47.116655 30019 hybrid_clock.cc:648] HybridClock initialized: now 1786016747116639 us; error 35 us; skew 500 ppm
Aug 06 11:45:47 dist-test-slave-45dx krb5kdc[29997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:47.119491 30019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:45:47.120488 30019 webserver.cc:460] Webserver started at http://127.0.0.1:33507/ using document root <none> and password file <none>
I20260806 11:45:47.121049 30019 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:47.121145 30019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:47.121387 30019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:47.123467 30019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "4a570b901df044a6a7d099709e155824"
format_stamp: "Formatted at 2026-08-06 11:45:47 on dist-test-slave-45dx"
server_key: "66a8895c7b96974947376ac9d2211a4e"
server_key_iv: "c63d0d00c3bdc4334967289f25b9bd39"
server_key_version: "encryptionkey@0"
I20260806 11:45:47.124033 30019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "4a570b901df044a6a7d099709e155824"
format_stamp: "Formatted at 2026-08-06 11:45:47 on dist-test-slave-45dx"
server_key: "66a8895c7b96974947376ac9d2211a4e"
server_key_iv: "c63d0d00c3bdc4334967289f25b9bd39"
server_key_version: "encryptionkey@0"
I20260806 11:45:47.127625 30019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 11:45:47.130000 30043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:47.131102 30019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:45:47.131244 30019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "4a570b901df044a6a7d099709e155824"
format_stamp: "Formatted at 2026-08-06 11:45:47 on dist-test-slave-45dx"
server_key: "66a8895c7b96974947376ac9d2211a4e"
server_key_iv: "c63d0d00c3bdc4334967289f25b9bd39"
server_key_version: "encryptionkey@0"
I20260806 11:45:47.131366 30019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:47.144724 30019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:47.147845 30019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:47.148070 30019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:47.155095 30019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39393
I20260806 11:45:47.155109 30111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39393 every 8 connection(s)
I20260806 11:45:47.155999 30019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:47.158181 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 30019
I20260806 11:45:47.158327 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:47.158654 452 external_mini_cluster.cc:1428] Setting key 4c82a37651bcbd636d1d40e3f80b3064
I20260806 11:45:47.158991 30113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:47.166644 30113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824: Bootstrap starting.
Aug 06 11:45:47 dist-test-slave-45dx krb5kdc[29997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 11:45:47.169415 30113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:47.170310 30113 log.cc:824] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:47.172770 30113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824: No bootstrap required, opened a new log
I20260806 11:45:47.174043 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:47.160040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41475 (local address 127.0.0.1:39393)
0806 11:45:47.160489 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:47.160505 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:47.160541 (+ 36us) server_negotiation.cc:407] Connection header received
0806 11:45:47.161205 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:47.161221 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:47.161544 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:47.161867 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:47.162964 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:47.163915 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:47.165202 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:47.165774 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:47.165983 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:47.168887 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:47.168892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:47.168905 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:47.168932 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:47.170895 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:47.171528 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:47.171532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:47.171537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:47.171644 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:47.172052 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:47.172054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:47.172055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:47.172518 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:47.172688 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:47.173085 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 11:45:47.173357 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":135,"threads_started":1}
I20260806 11:45:47.176697 30113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a570b901df044a6a7d099709e155824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39393 } }
I20260806 11:45:47.176959 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:45:47.176959 30113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:47.177023 30113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a570b901df044a6a7d099709e155824, State: Initialized, Role: FOLLOWER
I20260806 11:45:47.177489 30113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a570b901df044a6a7d099709e155824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39393 } }
I20260806 11:45:47.177670 30113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:47.177738 30113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:47.177856 30113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:47.179221 30113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a570b901df044a6a7d099709e155824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39393 } }
I20260806 11:45:47.179632 30113 leader_election.cc:302] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a570b901df044a6a7d099709e155824; no voters:
I20260806 11:45:47.180085 30113 leader_election.cc:288] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:47.180183 30126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:47.180495 30126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [term 1 LEADER]: Becoming Leader. State: Replica: 4a570b901df044a6a7d099709e155824, State: Running, Role: LEADER
I20260806 11:45:47.180810 30126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a570b901df044a6a7d099709e155824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39393 } }
I20260806 11:45:47.181579 30113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:47.181896 30126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a570b901df044a6a7d099709e155824. Latest consensus state: current_term: 1 leader_uuid: "4a570b901df044a6a7d099709e155824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a570b901df044a6a7d099709e155824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39393 } } }
I20260806 11:45:47.182027 30126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:47.182467 30132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:47.182467 30127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a570b901df044a6a7d099709e155824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a570b901df044a6a7d099709e155824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39393 } } }
I20260806 11:45:47.182718 30127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:47.185593 30132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:47.190157 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:47.179439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37751 (local address 127.0.0.1:39393)
0806 11:45:47.179557 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:47.179561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:47.179785 (+ 224us) server_negotiation.cc:407] Connection header received
0806 11:45:47.179907 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:47.179910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:47.179948 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:47.180013 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:47.180972 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:47.181816 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:47.183140 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:47.183592 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:47.183699 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:47.185662 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:47.185666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:47.185669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:47.185694 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:47.187687 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:47.188396 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:47.188400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:47.188402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:47.188502 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:47.188952 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:47.188959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:47.188961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:47.189191 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:47.189350 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:47.189891 (+ 541us) server_negotiation.cc:299] Negotiation successful
0806 11:45:47.189995 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 11:45:47.191284 30132 catalog_manager.cc:1347] Generated new cluster ID: 926a7661297f41609776b30b520a7689
I20260806 11:45:47.191354 30132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:47.198309 30132 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:47.199205 30132 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:47.212724 30132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a570b901df044a6a7d099709e155824: Generated new TSK 0
I20260806 11:45:47.213387 30132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:47.225260 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 30019
W20260806 11:45:47.243121 29995 connection.cc:535] client connection to 127.0.0.1:39393 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39393): Connection reset by peer (error 104)
2026-08-06T11:45:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1592 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:47 dist-test-slave-45dx krb5kdc[30154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:47 dist-test-slave-45dx krb5kdc[30154](info): set up 2 sockets
Aug 06 11:45:47 dist-test-slave-45dx krb5kdc[30154](info): commencing operation
krb5kdc: starting...
W20260806 11:45:48.492084 452 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:48 dist-test-slave-45dx krb5kdc[30154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:45:48.623731 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:48.755129 30173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:48.755410 30173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:48.755460 30173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:48.759474 30173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:48.759542 30173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:48.759562 30173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:48.759579 30173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:48.759594 30173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:48.764936 30173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:48.765965 30173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:48.766957 30173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:48.775781 30187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:48.775836 30184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:48.775861 30173 server_base.cc:1034] running on GCE node
W20260806 11:45:48.775781 30182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:48.776692 30173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:48.777776 30173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:48.778941 30173 hybrid_clock.cc:648] HybridClock initialized: now 1786016748778915 us; error 41 us; skew 500 ppm
Aug 06 11:45:48 dist-test-slave-45dx krb5kdc[30154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:48.781725 30173 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:45:48.782651 30173 webserver.cc:460] Webserver started at http://127.0.0.1:42965/ using document root <none> and password file <none>
I20260806 11:45:48.783207 30173 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:48.783300 30173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:48.783543 30173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:48.785616 30173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "5cd64ed0e0de497ba5492a61146341bd"
format_stamp: "Formatted at 2026-08-06 11:45:48 on dist-test-slave-45dx"
server_key: "3b614af30b11d19b561e4d595f8bef15"
server_key_iv: "5c2e258d305ea8b3a485b033328791f0"
server_key_version: "encryptionkey@0"
I20260806 11:45:48.786186 30173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "5cd64ed0e0de497ba5492a61146341bd"
format_stamp: "Formatted at 2026-08-06 11:45:48 on dist-test-slave-45dx"
server_key: "3b614af30b11d19b561e4d595f8bef15"
server_key_iv: "5c2e258d305ea8b3a485b033328791f0"
server_key_version: "encryptionkey@0"
I20260806 11:45:48.789803 30173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:45:48.792073 30198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:48.793301 30173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:45:48.793452 30173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "5cd64ed0e0de497ba5492a61146341bd"
format_stamp: "Formatted at 2026-08-06 11:45:48 on dist-test-slave-45dx"
server_key: "3b614af30b11d19b561e4d595f8bef15"
server_key_iv: "5c2e258d305ea8b3a485b033328791f0"
server_key_version: "encryptionkey@0"
I20260806 11:45:48.793570 30173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:48.815819 30173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:48.818670 30173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:48.818876 30173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:48.826057 30173 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38867
I20260806 11:45:48.826079 30274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38867 every 8 connection(s)
I20260806 11:45:48.827039 30173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:48.829843 30275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:48.834339 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 30173
I20260806 11:45:48.834496 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:48.834825 452 external_mini_cluster.cc:1428] Setting key 114b60d9213bfbb17c34677375a1c53f
I20260806 11:45:48.835605 30275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd: Bootstrap starting.
I20260806 11:45:48.837968 30275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:48.838732 30275 log.cc:824] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:48.840795 30275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd: No bootstrap required, opened a new log
I20260806 11:45:48.843887 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:48.836163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41991 (local address 127.0.0.1:38867)
0806 11:45:48.836606 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:48.836621 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:48.836656 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:45:48.837246 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:48.837261 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:48.837588 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:48.837848 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:48.838226 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:48.839223 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:48.840164 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:48.840861 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:48.841029 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:48.841863 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:48.841869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:48.841884 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:48.842381 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:48.842454 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:48.842483 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:48.842610 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:48.842785 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 11:45:48.842913 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":166,"threads_started":1}
I20260806 11:45:48.845261 30275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd64ed0e0de497ba5492a61146341bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } }
I20260806 11:45:48.845495 30275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:48.845587 30275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cd64ed0e0de497ba5492a61146341bd, State: Initialized, Role: FOLLOWER
I20260806 11:45:48.846026 30275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd64ed0e0de497ba5492a61146341bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } }
I20260806 11:45:48.846153 30275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:48.846238 30275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:48.846341 30275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:48.846884 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:45:48.847365 30275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd64ed0e0de497ba5492a61146341bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } }
I20260806 11:45:48.847694 30275 leader_election.cc:302] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cd64ed0e0de497ba5492a61146341bd; no voters:
I20260806 11:45:48.847982 30275 leader_election.cc:288] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:48.848146 30280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:48.848416 30280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [term 1 LEADER]: Becoming Leader. State: Replica: 5cd64ed0e0de497ba5492a61146341bd, State: Running, Role: LEADER
I20260806 11:45:48.848733 30280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd64ed0e0de497ba5492a61146341bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } }
I20260806 11:45:48.849323 30275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:48.850116 30280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cd64ed0e0de497ba5492a61146341bd. Latest consensus state: current_term: 1 leader_uuid: "5cd64ed0e0de497ba5492a61146341bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd64ed0e0de497ba5492a61146341bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } } }
I20260806 11:45:48.850237 30280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:48.850844 30286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cd64ed0e0de497ba5492a61146341bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd64ed0e0de497ba5492a61146341bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38867 } } }
I20260806 11:45:48.850956 30286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:48.851243 30298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:48.853932 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:48.849093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32771 (local address 127.0.0.1:38867)
0806 11:45:48.849208 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:48.849212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:48.849447 (+ 235us) server_negotiation.cc:407] Connection header received
0806 11:45:48.849599 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:48.849604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:48.849654 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:48.849746 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:48.850080 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:48.850919 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:48.852175 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:48.852499 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:48.852586 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:48.853436 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:48.853440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:48.853442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:48.853454 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:48.853480 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:48.853493 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:48.853577 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:48.853759 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 11:45:48.853794 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260806 11:45:48.855146 30298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:48.860806 30298 catalog_manager.cc:1347] Generated new cluster ID: 59364716e897406a947619918621c0eb
I20260806 11:45:48.860878 30298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:48.866366 30298 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:48.867406 30298 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:48.875336 30298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cd64ed0e0de497ba5492a61146341bd: Generated new TSK 0
I20260806 11:45:48.875958 30298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:48.880722 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 30173
W20260806 11:45:48.892886 30151 connection.cc:535] client connection to 127.0.0.1:38867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38867): Connection reset by peer (error 104)
2026-08-06T11:45:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1649 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:48 dist-test-slave-45dx krb5kdc[30318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:48 dist-test-slave-45dx krb5kdc[30318](info): set up 2 sockets
Aug 06 11:45:48 dist-test-slave-45dx krb5kdc[30318](info): commencing operation
krb5kdc: starting...
W20260806 11:45:50.126506 452 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:50 dist-test-slave-45dx krb5kdc[30318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:45:50.256819 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:50.369890 30333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:50.370198 30333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:50.370241 30333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:50.374563 30333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:50.374630 30333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:50.374655 30333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:50.374673 30333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:50.374688 30333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:50.380136 30333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:50.381278 30333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:50.382339 30333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:50.391155 30345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:50.391161 30344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:50.391216 30333 server_base.cc:1034] running on GCE node
W20260806 11:45:50.391194 30349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:50.391999 30333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:50.393210 30333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:50.394376 30333 hybrid_clock.cc:648] HybridClock initialized: now 1786016750394351 us; error 37 us; skew 500 ppm
Aug 06 11:45:50 dist-test-slave-45dx krb5kdc[30318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:50.402966 30333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:45:50.403914 30333 webserver.cc:460] Webserver started at http://127.0.0.1:33395/ using document root <none> and password file <none>
I20260806 11:45:50.404556 30333 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:50.404623 30333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:50.404882 30333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:50.406952 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "b3bc08b56fef4370a5252b90361434f3"
format_stamp: "Formatted at 2026-08-06 11:45:50 on dist-test-slave-45dx"
server_key: "329ac04d49faeba4cba3a2023ecd16b1"
server_key_iv: "c0cf69af2e47bfb3f602171db7a9a4f8"
server_key_version: "encryptionkey@0"
I20260806 11:45:50.407531 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "b3bc08b56fef4370a5252b90361434f3"
format_stamp: "Formatted at 2026-08-06 11:45:50 on dist-test-slave-45dx"
server_key: "329ac04d49faeba4cba3a2023ecd16b1"
server_key_iv: "c0cf69af2e47bfb3f602171db7a9a4f8"
server_key_version: "encryptionkey@0"
I20260806 11:45:50.411506 30333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 11:45:50.414232 30362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:50.415552 30333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:50.415714 30333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "b3bc08b56fef4370a5252b90361434f3"
format_stamp: "Formatted at 2026-08-06 11:45:50 on dist-test-slave-45dx"
server_key: "329ac04d49faeba4cba3a2023ecd16b1"
server_key_iv: "c0cf69af2e47bfb3f602171db7a9a4f8"
server_key_version: "encryptionkey@0"
I20260806 11:45:50.415839 30333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:50.420665 30333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:50.423703 30333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:50.423904 30333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:50.431638 30333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40633
I20260806 11:45:50.431665 30443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40633 every 8 connection(s)
I20260806 11:45:50.432626 30333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:50.433498 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 30333
I20260806 11:45:50.433661 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:50.433990 452 external_mini_cluster.cc:1428] Setting key 18b0ea6763d0c18ee189882814e73c9b
I20260806 11:45:50.435865 30445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:50.439733 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:50.435496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39623 (local address 127.0.0.1:40633)
0806 11:45:50.435949 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:50.435969 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:50.436012 (+ 43us) server_negotiation.cc:407] Connection header received
0806 11:45:50.436858 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:50.436881 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:50.437217 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:50.437584 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:50.438354 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:50.438360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:50.438375 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:50.438388 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:50.438454 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:50.438482 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:50.438626 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:50.438692 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 11:45:50.438840 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260806 11:45:50.442471 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:45:50.442817 30445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3: Bootstrap starting.
I20260806 11:45:50.445273 30445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:50.446228 30445 log.cc:824] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:50.446270 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:50.444599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43783 (local address 127.0.0.1:40633)
0806 11:45:50.444729 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:50.444733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:50.444936 (+ 203us) server_negotiation.cc:407] Connection header received
0806 11:45:50.445045 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:50.445048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:50.445077 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:50.445140 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:50.445850 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:50.445853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:50.445855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:50.445866 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:50.445891 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:50.445901 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:50.445970 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:50.446102 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 11:45:50.446132 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 11:45:50.448189 30445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3: No bootstrap required, opened a new log
I20260806 11:45:50.451674 30445 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3bc08b56fef4370a5252b90361434f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40633 } }
I20260806 11:45:50.451936 30445 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:50.452037 30445 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3bc08b56fef4370a5252b90361434f3, State: Initialized, Role: FOLLOWER
I20260806 11:45:50.452528 30445 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3bc08b56fef4370a5252b90361434f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40633 } }
I20260806 11:45:50.452684 30445 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:50.452757 30445 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:50.452868 30445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:50.453823 30445 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3bc08b56fef4370a5252b90361434f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40633 } }
I20260806 11:45:50.454175 30445 leader_election.cc:302] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3bc08b56fef4370a5252b90361434f3; no voters:
I20260806 11:45:50.454550 30445 leader_election.cc:288] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:50.454695 30460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:50.454974 30460 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [term 1 LEADER]: Becoming Leader. State: Replica: b3bc08b56fef4370a5252b90361434f3, State: Running, Role: LEADER
I20260806 11:45:50.455343 30460 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3bc08b56fef4370a5252b90361434f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40633 } }
I20260806 11:45:50.455875 30445 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:50.457163 30464 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3bc08b56fef4370a5252b90361434f3. Latest consensus state: current_term: 1 leader_uuid: "b3bc08b56fef4370a5252b90361434f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3bc08b56fef4370a5252b90361434f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40633 } } }
I20260806 11:45:50.457296 30464 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:50.457479 30461 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3bc08b56fef4370a5252b90361434f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3bc08b56fef4370a5252b90361434f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40633 } } }
I20260806 11:45:50.457628 30461 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:50.457746 30474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:50.460727 30474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:50.466581 30474 catalog_manager.cc:1347] Generated new cluster ID: 9e3e1a1cb7e54876bc024db47747dc4a
I20260806 11:45:50.466686 30474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:50.484588 30474 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:50.484984 30474 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:50.494378 30474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3bc08b56fef4370a5252b90361434f3: Generated new TSK 0
I20260806 11:45:50.495198 30474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:50.505312 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 30333
2026-08-06T11:45:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1626 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:50 dist-test-slave-45dx krb5kdc[30491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:50 dist-test-slave-45dx krb5kdc[30491](info): set up 2 sockets
Aug 06 11:45:50 dist-test-slave-45dx krb5kdc[30491](info): commencing operation
krb5kdc: starting...
W20260806 11:45:51.727100 452 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:51 dist-test-slave-45dx krb5kdc[30491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:45:51.860039 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:51.970290 30512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:51.970578 30512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:51.970664 30512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:51.974591 30512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:51.974684 30512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:51.974723 30512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:51.974782 30512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:51.974823 30512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:51.979986 30512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:51.981016 30512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:51.982023 30512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:51.990460 30521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:51.990448 30527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:51.990448 30520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:51.990758 30512 server_base.cc:1034] running on GCE node
I20260806 11:45:51.991362 30512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:51.992482 30512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:51.993661 30512 hybrid_clock.cc:648] HybridClock initialized: now 1786016751993641 us; error 32 us; skew 500 ppm
Aug 06 11:45:51 dist-test-slave-45dx krb5kdc[30491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:51.996497 30512 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:45:51.997407 30512 webserver.cc:460] Webserver started at http://127.0.0.1:33839/ using document root <none> and password file <none>
I20260806 11:45:51.997957 30512 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:51.998020 30512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:51.998260 30512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:52.000283 30512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "53ed67ae31494fedbf91233bcd869ac1"
format_stamp: "Formatted at 2026-08-06 11:45:51 on dist-test-slave-45dx"
server_key: "19fabce86e2aee28a7c6acb9d995a59d"
server_key_iv: "3e1e33c632626def9e33aa8ecd414826"
server_key_version: "encryptionkey@0"
I20260806 11:45:52.000926 30512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "53ed67ae31494fedbf91233bcd869ac1"
format_stamp: "Formatted at 2026-08-06 11:45:51 on dist-test-slave-45dx"
server_key: "19fabce86e2aee28a7c6acb9d995a59d"
server_key_iv: "3e1e33c632626def9e33aa8ecd414826"
server_key_version: "encryptionkey@0"
I20260806 11:45:52.004607 30512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:45:52.006989 30536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:52.008183 30512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:52.008311 30512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "53ed67ae31494fedbf91233bcd869ac1"
format_stamp: "Formatted at 2026-08-06 11:45:51 on dist-test-slave-45dx"
server_key: "19fabce86e2aee28a7c6acb9d995a59d"
server_key_iv: "3e1e33c632626def9e33aa8ecd414826"
server_key_version: "encryptionkey@0"
I20260806 11:45:52.008448 30512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:52.022949 30512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:52.026139 30512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:52.026369 30512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:52.033711 30512 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44943
I20260806 11:45:52.033754 30610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44943 every 8 connection(s)
I20260806 11:45:52.034811 30512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:52.036943 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 30512
I20260806 11:45:52.037113 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:52.037446 452 external_mini_cluster.cc:1428] Setting key 33d096c24400c4028dec8693f3bf8fb7
I20260806 11:45:52.038630 30611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:52.045996 30611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1: Bootstrap starting.
Aug 06 11:45:52 dist-test-slave-45dx krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 11:45:52.048521 30611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:52.049490 30611 log.cc:824] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:52.051528 30611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1: No bootstrap required, opened a new log
I20260806 11:45:52.053014 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:52.038846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35481 (local address 127.0.0.1:44943)
0806 11:45:52.039304 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:52.039334 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:52.039377 (+ 43us) server_negotiation.cc:407] Connection header received
0806 11:45:52.040259 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:52.040283 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:52.040759 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:52.041127 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:52.041859 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:52.043361 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:52.044230 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:52.045013 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:52.045174 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:52.047819 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:52.047825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:52.047842 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:52.047875 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:52.050079 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:52.050552 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:52.050558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:52.050564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:52.050689 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:52.050992 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:52.050995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:52.050997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:52.051416 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:52.051625 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:52.051922 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 11:45:52.052230 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260806 11:45:52.055418 30611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ed67ae31494fedbf91233bcd869ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } }
I20260806 11:45:52.055670 30611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:52.055766 30611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ed67ae31494fedbf91233bcd869ac1, State: Initialized, Role: FOLLOWER
I20260806 11:45:52.055905 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:45:52.056259 30611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ed67ae31494fedbf91233bcd869ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } }
I20260806 11:45:52.056491 30611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:52.056586 30611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:52.056689 30611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:52.057739 30611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ed67ae31494fedbf91233bcd869ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } }
I20260806 11:45:52.058146 30611 leader_election.cc:302] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53ed67ae31494fedbf91233bcd869ac1; no voters:
I20260806 11:45:52.058574 30611 leader_election.cc:288] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:52.058681 30626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:52.058951 30626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [term 1 LEADER]: Becoming Leader. State: Replica: 53ed67ae31494fedbf91233bcd869ac1, State: Running, Role: LEADER
I20260806 11:45:52.059340 30626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ed67ae31494fedbf91233bcd869ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } }
I20260806 11:45:52.059924 30611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:52.060534 30633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ed67ae31494fedbf91233bcd869ac1. Latest consensus state: current_term: 1 leader_uuid: "53ed67ae31494fedbf91233bcd869ac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ed67ae31494fedbf91233bcd869ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } } }
I20260806 11:45:52.060667 30633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:52.061129 30641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:52.062718 30630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53ed67ae31494fedbf91233bcd869ac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ed67ae31494fedbf91233bcd869ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } } }
I20260806 11:45:52.062855 30630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:52.064612 30641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:52.067591 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:52.058188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49489 (local address 127.0.0.1:44943)
0806 11:45:52.058368 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:52.058379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:52.058446 (+ 67us) server_negotiation.cc:407] Connection header received
0806 11:45:52.058618 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:52.058621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:52.058673 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:52.058762 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:52.059557 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:52.060376 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:52.061587 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:52.061939 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:52.062038 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:52.063848 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:52.063853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:52.063855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:52.063882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:52.065733 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:52.066292 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:52.066295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:52.066297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:52.066392 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:52.066689 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:52.066692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:52.066694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:52.066885 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:52.067035 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:52.067289 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 11:45:52.067431 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 11:45:52.070883 30641 catalog_manager.cc:1347] Generated new cluster ID: 54b73d12344b45fb9a2b931cb62281d2
I20260806 11:45:52.070960 30641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:52.086257 30641 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:52.087221 30641 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:52.099906 30641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53ed67ae31494fedbf91233bcd869ac1: Generated new TSK 0
I20260806 11:45:52.100693 30641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:52.113502 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 30512
2026-08-06T11:45:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1607 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:52 dist-test-slave-45dx krb5kdc[30659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:52 dist-test-slave-45dx krb5kdc[30659](info): set up 2 sockets
Aug 06 11:45:52 dist-test-slave-45dx krb5kdc[30659](info): commencing operation
krb5kdc: starting...
W20260806 11:45:53.336575 452 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:53 dist-test-slave-45dx krb5kdc[30659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:45:53.464483 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:53.577957 30677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:53.578253 30677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:53.578295 30677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:53.582379 30677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:53.582566 30677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:53.582618 30677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:53.582638 30677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:53.582679 30677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:53.588160 30677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:53.589282 30677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:53.590318 30677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 11:45:53.599228 30677 server_base.cc:1034] running on GCE node
W20260806 11:45:53.599190 30688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:53.599180 30693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:53.599190 30691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:53.600093 30677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:53.601263 30677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:53.602474 30677 hybrid_clock.cc:648] HybridClock initialized: now 1786016753602443 us; error 44 us; skew 500 ppm
Aug 06 11:45:53 dist-test-slave-45dx krb5kdc[30659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:53.605444 30677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:45:53.606386 30677 webserver.cc:460] Webserver started at http://127.0.0.1:40399/ using document root <none> and password file <none>
I20260806 11:45:53.606949 30677 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:53.607043 30677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:53.607280 30677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:53.609369 30677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "37038b3e8e2240db9679ebe8a2713cb1"
format_stamp: "Formatted at 2026-08-06 11:45:53 on dist-test-slave-45dx"
server_key: "6a00b682695ded24d5bf26801af06553"
server_key_iv: "84086e8fd81daef10117057e20077d38"
server_key_version: "encryptionkey@0"
I20260806 11:45:53.609942 30677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "37038b3e8e2240db9679ebe8a2713cb1"
format_stamp: "Formatted at 2026-08-06 11:45:53 on dist-test-slave-45dx"
server_key: "6a00b682695ded24d5bf26801af06553"
server_key_iv: "84086e8fd81daef10117057e20077d38"
server_key_version: "encryptionkey@0"
I20260806 11:45:53.613646 30677 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:45:53.616087 30703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:53.617491 30677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:45:53.617650 30677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "37038b3e8e2240db9679ebe8a2713cb1"
format_stamp: "Formatted at 2026-08-06 11:45:53 on dist-test-slave-45dx"
server_key: "6a00b682695ded24d5bf26801af06553"
server_key_iv: "84086e8fd81daef10117057e20077d38"
server_key_version: "encryptionkey@0"
I20260806 11:45:53.617774 30677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:53.633634 30677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:53.637075 30677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:53.637300 30677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:53.644761 30677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40475
I20260806 11:45:53.644788 30784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40475 every 8 connection(s)
I20260806 11:45:53.645714 30677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:53.648470 30785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:53.651784 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 30677
I20260806 11:45:53.651960 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:53.652261 452 external_mini_cluster.cc:1428] Setting key 402a9ca84377c70eff950caa30da4f79
I20260806 11:45:53.654803 30785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1: Bootstrap starting.
I20260806 11:45:53.657511 30785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:53.658329 30785 log.cc:824] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:53.660641 30785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1: No bootstrap required, opened a new log
I20260806 11:45:53.662410 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:53.653733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48117 (local address 127.0.0.1:40475)
0806 11:45:53.654193 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:53.654214 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:53.654257 (+ 43us) server_negotiation.cc:407] Connection header received
0806 11:45:53.655103 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:53.655125 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:53.655553 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:53.655926 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:53.656399 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:53.657880 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:53.658823 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:53.659422 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:53.659575 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:53.660411 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:53.660418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:53.660435 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:53.660935 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:53.661010 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:53.661042 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:53.661144 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:53.661265 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 11:45:53.661470 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":123,"threads_started":1}
I20260806 11:45:53.665134 30785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37038b3e8e2240db9679ebe8a2713cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40475 } }
I20260806 11:45:53.665396 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:45:53.665388 30785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:53.665462 30785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37038b3e8e2240db9679ebe8a2713cb1, State: Initialized, Role: FOLLOWER
I20260806 11:45:53.665916 30785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37038b3e8e2240db9679ebe8a2713cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40475 } }
I20260806 11:45:53.666040 30785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:53.666124 30785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:53.666234 30785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:53.667212 30785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37038b3e8e2240db9679ebe8a2713cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40475 } }
I20260806 11:45:53.667593 30785 leader_election.cc:302] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37038b3e8e2240db9679ebe8a2713cb1; no voters:
I20260806 11:45:53.667951 30785 leader_election.cc:288] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:53.668084 30798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:53.668342 30798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [term 1 LEADER]: Becoming Leader. State: Replica: 37038b3e8e2240db9679ebe8a2713cb1, State: Running, Role: LEADER
I20260806 11:45:53.668679 30798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37038b3e8e2240db9679ebe8a2713cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40475 } }
I20260806 11:45:53.669214 30785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:53.669893 30798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37038b3e8e2240db9679ebe8a2713cb1. Latest consensus state: current_term: 1 leader_uuid: "37038b3e8e2240db9679ebe8a2713cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37038b3e8e2240db9679ebe8a2713cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40475 } } }
I20260806 11:45:53.670006 30798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:53.671271 30805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37038b3e8e2240db9679ebe8a2713cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37038b3e8e2240db9679ebe8a2713cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40475 } } }
I20260806 11:45:53.671442 30805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:53.672676 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:53.667575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54997 (local address 127.0.0.1:40475)
0806 11:45:53.667708 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:53.667713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:53.667977 (+ 264us) server_negotiation.cc:407] Connection header received
0806 11:45:53.668120 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:53.668124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:53.668177 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:53.668271 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:53.668716 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:53.669483 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:53.670681 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:53.671028 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:53.671127 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:53.672153 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:53.672159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:53.672162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:53.672173 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:53.672201 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:53.672212 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:53.672279 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:53.672497 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 11:45:53.672552 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 11:45:53.674780 30824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:45:53.674847 30824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:45:53.674938 30812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:53.675664 30812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:53.680982 30812 catalog_manager.cc:1347] Generated new cluster ID: 233b849984b34db9aaa4468d3e0d638b
I20260806 11:45:53.681042 30812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:53.685577 30812 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:53.686734 30812 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:53.696719 30812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37038b3e8e2240db9679ebe8a2713cb1: Generated new TSK 0
I20260806 11:45:53.697484 30812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:53.707044 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 30677
2026-08-06T11:45:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1594 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:53 dist-test-slave-45dx krb5kdc[30830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:53 dist-test-slave-45dx krb5kdc[30830](info): set up 2 sockets
Aug 06 11:45:53 dist-test-slave-45dx krb5kdc[30830](info): commencing operation
krb5kdc: starting...
W20260806 11:45:54.937911 452 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:55 dist-test-slave-45dx krb5kdc[30830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:45:55.067183 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:55.177572 30855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:55.177801 30855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:55.177837 30855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:55.181856 30855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:55.181921 30855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:55.181941 30855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:55.181957 30855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:55.182010 30855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:55.187321 30855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:55.188362 30855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:55.189378 30855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:55.197383 30864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:55.197407 30862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:55.197527 30855 server_base.cc:1034] running on GCE node
W20260806 11:45:55.197396 30861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:55.198284 30855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:55.199360 30855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:55.200548 30855 hybrid_clock.cc:648] HybridClock initialized: now 1786016755200529 us; error 34 us; skew 500 ppm
Aug 06 11:45:55 dist-test-slave-45dx krb5kdc[30830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:55.203320 30855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:45:55.204108 30855 webserver.cc:460] Webserver started at http://127.0.0.1:40717/ using document root <none> and password file <none>
I20260806 11:45:55.204694 30855 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:55.204757 30855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:55.204922 30855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:55.206804 30855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "7958a7223f484e04862bb3e236c1a190"
format_stamp: "Formatted at 2026-08-06 11:45:55 on dist-test-slave-45dx"
server_key: "21ad7cb898028a786f8c6449c7da6ddb"
server_key_iv: "e43c2f87e689b35ca4a6660cee75f573"
server_key_version: "encryptionkey@0"
I20260806 11:45:55.207307 30855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "7958a7223f484e04862bb3e236c1a190"
format_stamp: "Formatted at 2026-08-06 11:45:55 on dist-test-slave-45dx"
server_key: "21ad7cb898028a786f8c6449c7da6ddb"
server_key_iv: "e43c2f87e689b35ca4a6660cee75f573"
server_key_version: "encryptionkey@0"
I20260806 11:45:55.210810 30855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 11:45:55.213055 30876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:55.214247 30855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:45:55.214346 30855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "7958a7223f484e04862bb3e236c1a190"
format_stamp: "Formatted at 2026-08-06 11:45:55 on dist-test-slave-45dx"
server_key: "21ad7cb898028a786f8c6449c7da6ddb"
server_key_iv: "e43c2f87e689b35ca4a6660cee75f573"
server_key_version: "encryptionkey@0"
I20260806 11:45:55.214427 30855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:55.218743 30855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:55.221447 30855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:55.221586 30855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:55.228394 30855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33987
I20260806 11:45:55.228446 30953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33987 every 8 connection(s)
I20260806 11:45:55.229282 30855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:55.231866 30954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:55.233392 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 30855
I20260806 11:45:55.233554 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:55.233879 452 external_mini_cluster.cc:1428] Setting key 0b875692b228a05245a64e63edf047f1
I20260806 11:45:55.238030 30954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190: Bootstrap starting.
I20260806 11:45:55.238893 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:55.235259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37447 (local address 127.0.0.1:33987)
0806 11:45:55.235677 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:55.235695 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:55.235728 (+ 33us) server_negotiation.cc:407] Connection header received
0806 11:45:55.236443 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:55.236460 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:55.236716 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:55.237028 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:55.237801 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:55.237805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:55.237817 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:55.237827 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:55.237880 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:55.237899 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:55.238059 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:55.238127 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 11:45:55.238271 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":147,"threads_started":1}
I20260806 11:45:55.240480 30954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:55.241289 30954 log.cc:824] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:55.241603 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:45:55.243278 30954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190: No bootstrap required, opened a new log
I20260806 11:45:55.245198 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:55.243645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32903 (local address 127.0.0.1:33987)
0806 11:45:55.243758 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:55.243762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:55.243948 (+ 186us) server_negotiation.cc:407] Connection header received
0806 11:45:55.244031 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:55.244034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:55.244065 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:55.244169 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:55.244840 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:55.244843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:55.244846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:55.244857 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:55.244893 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:55.244907 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:55.244994 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:55.245042 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 11:45:55.245074 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 11:45:55.247607 30954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958a7223f484e04862bb3e236c1a190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } }
I20260806 11:45:55.247825 30954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:55.247896 30954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958a7223f484e04862bb3e236c1a190, State: Initialized, Role: FOLLOWER
I20260806 11:45:55.248296 30954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958a7223f484e04862bb3e236c1a190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } }
I20260806 11:45:55.248481 30954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:55.248562 30954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:55.248667 30954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:55.249631 30954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958a7223f484e04862bb3e236c1a190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } }
I20260806 11:45:55.249912 30954 leader_election.cc:302] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7958a7223f484e04862bb3e236c1a190; no voters:
I20260806 11:45:55.250154 30954 leader_election.cc:288] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:55.250272 30969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:55.250550 30969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [term 1 LEADER]: Becoming Leader. State: Replica: 7958a7223f484e04862bb3e236c1a190, State: Running, Role: LEADER
I20260806 11:45:55.250878 30969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958a7223f484e04862bb3e236c1a190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } }
I20260806 11:45:55.251367 30954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:55.252552 30974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7958a7223f484e04862bb3e236c1a190. Latest consensus state: current_term: 1 leader_uuid: "7958a7223f484e04862bb3e236c1a190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958a7223f484e04862bb3e236c1a190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } } }
I20260806 11:45:55.252722 30974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:55.253031 30970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7958a7223f484e04862bb3e236c1a190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958a7223f484e04862bb3e236c1a190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } } }
I20260806 11:45:55.253187 30982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:55.253120 30970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:55.256016 30982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:55.261123 30982 catalog_manager.cc:1347] Generated new cluster ID: 8fa4435783b645d782e0358f0f62a5ce
I20260806 11:45:55.261179 30982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:55.283371 30982 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:55.283727 30982 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:55.298054 30982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7958a7223f484e04862bb3e236c1a190: Generated new TSK 0
I20260806 11:45:55.298859 30982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:55.313206 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 30855
2026-08-06T11:45:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1603 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:55 dist-test-slave-45dx krb5kdc[30997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:55 dist-test-slave-45dx krb5kdc[30997](info): set up 2 sockets
Aug 06 11:45:55 dist-test-slave-45dx krb5kdc[30997](info): commencing operation
krb5kdc: starting...
W20260806 11:45:56.533075 452 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:56 dist-test-slave-45dx krb5kdc[30997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:45:56.657073 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:56.767549 31017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:56.767848 31017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:56.767938 31017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:56.771921 31017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:56.772011 31017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:56.772046 31017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:56.772085 31017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:56.772116 31017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:56.777323 31017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:56.778393 31017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:56.779425 31017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:56.788542 31028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:56.788551 31026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:56.788563 31030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:56.788945 31017 server_base.cc:1034] running on GCE node
I20260806 11:45:56.789542 31017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:56.790764 31017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:56.791950 31017 hybrid_clock.cc:648] HybridClock initialized: now 1786016756791924 us; error 40 us; skew 500 ppm
Aug 06 11:45:56 dist-test-slave-45dx krb5kdc[30997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:56.799613 31017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:45:56.800549 31017 webserver.cc:460] Webserver started at http://127.0.0.1:41819/ using document root <none> and password file <none>
I20260806 11:45:56.801126 31017 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:56.801221 31017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:56.801458 31017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:56.803445 31017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "3bfcd9f327bb4248b6ca5195e31dc867"
format_stamp: "Formatted at 2026-08-06 11:45:56 on dist-test-slave-45dx"
server_key: "2e2a27dfd7616d00bf26c24dd9143e7a"
server_key_iv: "926d0870cb8533e72f1557269ada2478"
server_key_version: "encryptionkey@0"
I20260806 11:45:56.804006 31017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "3bfcd9f327bb4248b6ca5195e31dc867"
format_stamp: "Formatted at 2026-08-06 11:45:56 on dist-test-slave-45dx"
server_key: "2e2a27dfd7616d00bf26c24dd9143e7a"
server_key_iv: "926d0870cb8533e72f1557269ada2478"
server_key_version: "encryptionkey@0"
I20260806 11:45:56.807610 31017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:45:56.810010 31040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:56.811164 31017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:45:56.811300 31017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "3bfcd9f327bb4248b6ca5195e31dc867"
format_stamp: "Formatted at 2026-08-06 11:45:56 on dist-test-slave-45dx"
server_key: "2e2a27dfd7616d00bf26c24dd9143e7a"
server_key_iv: "926d0870cb8533e72f1557269ada2478"
server_key_version: "encryptionkey@0"
I20260806 11:45:56.811415 31017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:56.849200 31017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:56.852042 31017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:56.852242 31017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:56.859294 31017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33087
I20260806 11:45:56.859323 31116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33087 every 8 connection(s)
I20260806 11:45:56.860229 31017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:56.863029 31117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:56.863559 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 31017
I20260806 11:45:56.863710 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:56.864007 452 external_mini_cluster.cc:1428] Setting key 04000df5fd4b472a950ce867f33e1450
I20260806 11:45:56.869004 31117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867: Bootstrap starting.
I20260806 11:45:56.871732 31117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:56.872742 31117 log.cc:824] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867: Log is configured to *not* fsync() on all Append() calls
Aug 06 11:45:56 dist-test-slave-45dx krb5kdc[30997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 11:45:56.874780 31117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867: No bootstrap required, opened a new log
I20260806 11:45:56.878418 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:56.865405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:34689 (local address 127.0.0.1:33087)
0806 11:45:56.865846 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:56.865867 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:56.865900 (+ 33us) server_negotiation.cc:407] Connection header received
0806 11:45:56.866551 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:56.866566 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:56.866885 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:56.867201 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:56.868046 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:56.869074 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:56.870291 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:56.870733 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:56.870862 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:56.873881 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:56.873886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:56.873899 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:56.873925 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:56.875601 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:56.876111 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:56.876116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:56.876121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:56.876229 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:56.876606 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:56.876609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:56.876611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:56.876989 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:56.877206 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:56.877520 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 11:45:56.877728 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260806 11:45:56.878356 31117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcd9f327bb4248b6ca5195e31dc867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33087 } }
I20260806 11:45:56.878600 31117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:56.878685 31117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfcd9f327bb4248b6ca5195e31dc867, State: Initialized, Role: FOLLOWER
I20260806 11:45:56.879168 31117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcd9f327bb4248b6ca5195e31dc867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33087 } }
I20260806 11:45:56.879297 31117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:56.879420 31117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:56.879544 31117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:56.880513 31117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcd9f327bb4248b6ca5195e31dc867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33087 } }
I20260806 11:45:56.880898 31117 leader_election.cc:302] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bfcd9f327bb4248b6ca5195e31dc867; no voters:
I20260806 11:45:56.881278 31117 leader_election.cc:288] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:56.881376 31124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:56.881422 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:45:56.881624 31124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [term 1 LEADER]: Becoming Leader. State: Replica: 3bfcd9f327bb4248b6ca5195e31dc867, State: Running, Role: LEADER
I20260806 11:45:56.882026 31124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcd9f327bb4248b6ca5195e31dc867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33087 } }
I20260806 11:45:56.882555 31117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:56.883689 31125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bfcd9f327bb4248b6ca5195e31dc867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcd9f327bb4248b6ca5195e31dc867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33087 } } }
I20260806 11:45:56.883836 31125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:56.884084 31125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bfcd9f327bb4248b6ca5195e31dc867. Latest consensus state: current_term: 1 leader_uuid: "3bfcd9f327bb4248b6ca5195e31dc867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcd9f327bb4248b6ca5195e31dc867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33087 } } }
I20260806 11:45:56.884176 31125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:56.885203 31151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:56.888574 31151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:56.892910 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:56.884390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:59629 (local address 127.0.0.1:33087)
0806 11:45:56.884558 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:56.884562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:56.884575 (+ 13us) server_negotiation.cc:407] Connection header received
0806 11:45:56.884617 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:56.884619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:56.884686 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:56.884775 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:56.885610 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:56.886323 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:56.887160 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:56.887413 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:56.887529 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:56.889073 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:56.889077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:56.889079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:45:56.889104 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:56.890730 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:56.891411 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:56.891414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:56.891416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:56.891508 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:45:56.891885 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:45:56.891888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:45:56.891889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:45:56.892096 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:56.892257 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:56.892652 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 11:45:56.892751 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 11:45:56.894407 31151 catalog_manager.cc:1347] Generated new cluster ID: b9d680101f5f449e96aeadd383769af8
I20260806 11:45:56.894470 31151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:56.917560 31151 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:56.918414 31151 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:56.923900 31151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bfcd9f327bb4248b6ca5195e31dc867: Generated new TSK 0
I20260806 11:45:56.924561 31151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:56.931358 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 31017
W20260806 11:45:56.944762 30995 connection.cc:535] client connection to 127.0.0.1:33087 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33087): Connection reset by peer (error 104)
2026-08-06T11:45:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1618 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:56 dist-test-slave-45dx krb5kdc[31166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:56 dist-test-slave-45dx krb5kdc[31166](info): set up 2 sockets
Aug 06 11:45:56 dist-test-slave-45dx krb5kdc[31166](info): commencing operation
krb5kdc: starting...
W20260806 11:45:58.140525 452 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:58 dist-test-slave-45dx krb5kdc[31166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:45:58.267503 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:45:58.380079 31181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:45:58.380401 31181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:45:58.380515 31181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:45:58.384487 31181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:45:58.384574 31181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:45:58.384611 31181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:45:58.384649 31181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:45:58.384677 31181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:45:58.389770 31181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:45:58.390770 31181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:45:58.391752 31181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:45:58.400106 31187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:58.400133 31188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:45:58.400115 31192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:45:58.400496 31181 server_base.cc:1034] running on GCE node
I20260806 11:45:58.401194 31181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:45:58.402274 31181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:45:58.403491 31181 hybrid_clock.cc:648] HybridClock initialized: now 1786016758403475 us; error 34 us; skew 500 ppm
Aug 06 11:45:58 dist-test-slave-45dx krb5kdc[31166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:45:58.406383 31181 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:45:58.407212 31181 webserver.cc:460] Webserver started at http://127.0.0.1:41779/ using document root <none> and password file <none>
I20260806 11:45:58.407740 31181 fs_manager.cc:359] Metadata directory not provided
I20260806 11:45:58.407796 31181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:45:58.407958 31181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:45:58.410002 31181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "0628a11f77314187b4c33de4570b15d2"
format_stamp: "Formatted at 2026-08-06 11:45:58 on dist-test-slave-45dx"
server_key: "ce8645f05ffa27c268bccd72f574c199"
server_key_iv: "722e750baf57a69628fd624e3d3cd79b"
server_key_version: "encryptionkey@0"
I20260806 11:45:58.410529 31181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "0628a11f77314187b4c33de4570b15d2"
format_stamp: "Formatted at 2026-08-06 11:45:58 on dist-test-slave-45dx"
server_key: "ce8645f05ffa27c268bccd72f574c199"
server_key_iv: "722e750baf57a69628fd624e3d3cd79b"
server_key_version: "encryptionkey@0"
I20260806 11:45:58.414125 31181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 11:45:58.416503 31200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:45:58.417676 31181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 11:45:58.417810 31181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "0628a11f77314187b4c33de4570b15d2"
format_stamp: "Formatted at 2026-08-06 11:45:58 on dist-test-slave-45dx"
server_key: "ce8645f05ffa27c268bccd72f574c199"
server_key_iv: "722e750baf57a69628fd624e3d3cd79b"
server_key_version: "encryptionkey@0"
I20260806 11:45:58.417917 31181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:45:58.440892 31181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:45:58.444438 31181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:45:58.444633 31181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:45:58.451761 31181 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44333
I20260806 11:45:58.451788 31275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44333 every 8 connection(s)
I20260806 11:45:58.452749 31181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:45:58.455364 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 31181
I20260806 11:45:58.455492 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:45:58.455755 452 external_mini_cluster.cc:1428] Setting key e4ac6fda75d00de84296e758df5eebb3
I20260806 11:45:58.456038 31276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:45:58.462828 31276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2: Bootstrap starting.
I20260806 11:45:58.464582 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:58.457059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:32907 (local address 127.0.0.1:44333)
0806 11:45:58.457458 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:58.457477 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:58.457525 (+ 48us) server_negotiation.cc:407] Connection header received
0806 11:45:58.458151 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:58.458166 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:58.458484 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:58.458808 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:58.459164 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:58.460111 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:58.461332 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:58.461789 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:58.461936 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:58.462809 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:58.462814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:58.462825 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:58.463259 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:58.463313 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:58.463336 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:58.463439 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:58.463680 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 11:45:58.463820 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":148,"threads_started":1}
I20260806 11:45:58.465138 31276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2: Neither blocks nor log segments found. Creating new log.
I20260806 11:45:58.465993 31276 log.cc:824] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2: Log is configured to *not* fsync() on all Append() calls
I20260806 11:45:58.467549 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:45:58.468091 31276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2: No bootstrap required, opened a new log
I20260806 11:45:58.472214 31276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0628a11f77314187b4c33de4570b15d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260806 11:45:58.472504 31276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:45:58.472579 31276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0628a11f77314187b4c33de4570b15d2, State: Initialized, Role: FOLLOWER
I20260806 11:45:58.473059 31276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0628a11f77314187b4c33de4570b15d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260806 11:45:58.473223 31276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:45:58.473299 31276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:45:58.473420 31276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:45:58.474035 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:45:58.469858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:41001 (local address 127.0.0.1:44333)
0806 11:45:58.469976 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 11:45:58.469981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:45:58.470122 (+ 141us) server_negotiation.cc:407] Connection header received
0806 11:45:58.470199 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:45:58.470203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:45:58.470257 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:45:58.470353 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:45:58.470590 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:58.471325 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:58.472195 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:45:58.472565 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:45:58.472675 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:45:58.473577 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:45:58.473584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:45:58.473588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:45:58.473600 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:45:58.473628 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:45:58.473645 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:45:58.473744 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 11:45:58.473840 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 11:45:58.473899 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 11:45:58.474570 31276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0628a11f77314187b4c33de4570b15d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260806 11:45:58.474956 31276 leader_election.cc:302] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0628a11f77314187b4c33de4570b15d2; no voters:
I20260806 11:45:58.475302 31276 leader_election.cc:288] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:45:58.475416 31292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:45:58.475672 31292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [term 1 LEADER]: Becoming Leader. State: Replica: 0628a11f77314187b4c33de4570b15d2, State: Running, Role: LEADER
I20260806 11:45:58.475991 31292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0628a11f77314187b4c33de4570b15d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260806 11:45:58.476624 31276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:45:58.477699 31294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0628a11f77314187b4c33de4570b15d2. Latest consensus state: current_term: 1 leader_uuid: "0628a11f77314187b4c33de4570b15d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0628a11f77314187b4c33de4570b15d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } } }
I20260806 11:45:58.477834 31294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:58.478125 31293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0628a11f77314187b4c33de4570b15d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0628a11f77314187b4c33de4570b15d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } } }
I20260806 11:45:58.478233 31293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2 [sys.catalog]: This master's current role is: LEADER
I20260806 11:45:58.478886 31306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:45:58.481686 31306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:45:58.487469 31306 catalog_manager.cc:1347] Generated new cluster ID: cae909034cbe492aab61705823db436a
I20260806 11:45:58.487550 31306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:45:58.496387 31306 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:45:58.497596 31306 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:45:58.503403 31306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0628a11f77314187b4c33de4570b15d2: Generated new TSK 0
I20260806 11:45:58.504040 31306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:45:58.511649 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 31181
W20260806 11:45:58.533936 31163 connection.cc:535] client connection to 127.0.0.1:44333 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44333): Connection reset by peer (error 104)
2026-08-06T11:45:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1589 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:45:58 dist-test-slave-45dx krb5kdc[31328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:45:58 dist-test-slave-45dx krb5kdc[31328](info): set up 2 sockets
Aug 06 11:45:58 dist-test-slave-45dx krb5kdc[31328](info): commencing operation
krb5kdc: starting...
W20260806 11:45:59.757474 452 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:45:59 dist-test-slave-45dx krb5kdc[31328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:45:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:45:59.892977 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:00.015635 31352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:00.015965 31352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:00.016010 31352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:00.020197 31352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:00.020277 31352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:00.020298 31352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:00.020393 31352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:00.020432 31352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:00.026594 31352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:00.027753 31352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:00.028878 31352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:00.038069 31368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:00.038080 31366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:00.038085 31365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:00.038395 31352 server_base.cc:1034] running on GCE node
I20260806 11:46:00.039153 31352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:00.040274 31352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:00.041585 31352 hybrid_clock.cc:648] HybridClock initialized: now 1786016760041559 us; error 39 us; skew 500 ppm
Aug 06 11:46:00 dist-test-slave-45dx krb5kdc[31328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:00.044629 31352 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:46:00.045665 31352 webserver.cc:460] Webserver started at http://127.0.0.1:37113/ using document root <none> and password file <none>
I20260806 11:46:00.046268 31352 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:00.046336 31352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:00.046568 31352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:00.048844 31352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "5491fa53ebff469493d2ab3ba9acd119"
format_stamp: "Formatted at 2026-08-06 11:46:00 on dist-test-slave-45dx"
server_key: "81f51e770a0e4c74585330b51b6f15bc"
server_key_iv: "a14eec0f0e51ea1053625b5df94803a7"
server_key_version: "encryptionkey@0"
I20260806 11:46:00.049474 31352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "5491fa53ebff469493d2ab3ba9acd119"
format_stamp: "Formatted at 2026-08-06 11:46:00 on dist-test-slave-45dx"
server_key: "81f51e770a0e4c74585330b51b6f15bc"
server_key_iv: "a14eec0f0e51ea1053625b5df94803a7"
server_key_version: "encryptionkey@0"
I20260806 11:46:00.053325 31352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 11:46:00.055953 31384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:00.057292 31352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:46:00.057458 31352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "5491fa53ebff469493d2ab3ba9acd119"
format_stamp: "Formatted at 2026-08-06 11:46:00 on dist-test-slave-45dx"
server_key: "81f51e770a0e4c74585330b51b6f15bc"
server_key_iv: "a14eec0f0e51ea1053625b5df94803a7"
server_key_version: "encryptionkey@0"
I20260806 11:46:00.057583 31352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:00.062341 31352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:00.065603 31352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:00.065814 31352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:00.074551 31352 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44657
I20260806 11:46:00.074568 31464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44657 every 8 connection(s)
I20260806 11:46:00.075696 31352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:00.079080 31466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:00.081606 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 31352
I20260806 11:46:00.081785 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:00.082098 452 external_mini_cluster.cc:1428] Setting key abdf345d2024665e72791a9f31453f96
I20260806 11:46:00.086808 31466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119: Bootstrap starting.
I20260806 11:46:00.087661 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:00.083600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:49041 (local address 127.0.0.1:44657)
0806 11:46:00.084040 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:00.084065 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:00.084112 (+ 47us) server_negotiation.cc:407] Connection header received
0806 11:46:00.084855 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:00.084875 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:00.085153 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:00.085433 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:00.086370 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:00.086376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:00.086388 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:00.086399 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:00.086458 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:00.086479 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:00.086601 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:00.086729 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 11:46:00.086930 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":152,"threads_started":1}
I20260806 11:46:00.089180 31466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:00.090008 31466 log.cc:824] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:00.090553 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:00.092020 31466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119: No bootstrap required, opened a new log
I20260806 11:46:00.094434 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:00.092685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:42413 (local address 127.0.0.1:44657)
0806 11:46:00.092820 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:00.092824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:00.093017 (+ 193us) server_negotiation.cc:407] Connection header received
0806 11:46:00.093149 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:00.093151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:00.093178 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:00.093241 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:00.094024 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:00.094027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:00.094029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:00.094039 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:00.094066 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:00.094077 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:00.094150 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:00.094268 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 11:46:00.094300 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 11:46:00.096769 31466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5491fa53ebff469493d2ab3ba9acd119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } }
I20260806 11:46:00.097030 31466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:00.097114 31466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5491fa53ebff469493d2ab3ba9acd119, State: Initialized, Role: FOLLOWER
I20260806 11:46:00.097561 31466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5491fa53ebff469493d2ab3ba9acd119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } }
I20260806 11:46:00.097720 31466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:00.097822 31466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:00.097930 31466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:00.098982 31466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5491fa53ebff469493d2ab3ba9acd119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } }
I20260806 11:46:00.099361 31466 leader_election.cc:302] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5491fa53ebff469493d2ab3ba9acd119; no voters:
I20260806 11:46:00.099656 31466 leader_election.cc:288] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:00.099759 31483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:00.100028 31483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [term 1 LEADER]: Becoming Leader. State: Replica: 5491fa53ebff469493d2ab3ba9acd119, State: Running, Role: LEADER
I20260806 11:46:00.100374 31483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5491fa53ebff469493d2ab3ba9acd119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } }
I20260806 11:46:00.100972 31466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:00.102195 31484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5491fa53ebff469493d2ab3ba9acd119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5491fa53ebff469493d2ab3ba9acd119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } } }
I20260806 11:46:00.102339 31484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:00.102212 31485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5491fa53ebff469493d2ab3ba9acd119. Latest consensus state: current_term: 1 leader_uuid: "5491fa53ebff469493d2ab3ba9acd119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5491fa53ebff469493d2ab3ba9acd119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } } }
I20260806 11:46:00.102425 31485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119 [sys.catalog]: This master's current role is: LEADER
W20260806 11:46:00.106076 31504 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:00.106151 31504 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:00.106237 31494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:00.106940 31494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:00.112720 31494 catalog_manager.cc:1347] Generated new cluster ID: a6708837450d42dc9196d8ea4ca6bda4
I20260806 11:46:00.112797 31494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:00.157924 31494 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:00.158293 31494 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:00.167708 31494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5491fa53ebff469493d2ab3ba9acd119: Generated new TSK 0
I20260806 11:46:00.168476 31494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:00.177135 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 31352
2026-08-06T11:46:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1660 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:00 dist-test-slave-45dx krb5kdc[31511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:00 dist-test-slave-45dx krb5kdc[31511](info): set up 2 sockets
Aug 06 11:46:00 dist-test-slave-45dx krb5kdc[31511](info): commencing operation
krb5kdc: starting...
W20260806 11:46:01.417799 452 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:01 dist-test-slave-45dx krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:46:01.540998 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:01.651419 31528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:01.651731 31528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:01.651810 31528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:01.655858 31528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:01.655953 31528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:01.655992 31528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:01.656054 31528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:01.656095 31528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:01.661358 31528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:01.662407 31528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:01.663424 31528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:01.671751 31534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:01.671820 31540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:01.671811 31528 server_base.cc:1034] running on GCE node
W20260806 11:46:01.671741 31535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:01.672616 31528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:01.673712 31528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:01.674901 31528 hybrid_clock.cc:648] HybridClock initialized: now 1786016761674870 us; error 43 us; skew 500 ppm
Aug 06 11:46:01 dist-test-slave-45dx krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:01.677680 31528 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:46:01.678791 31528 webserver.cc:460] Webserver started at http://127.0.0.1:43801/ using document root <none> and password file <none>
I20260806 11:46:01.679356 31528 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:01.679455 31528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:01.679666 31528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:01.681726 31528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "6c7b851e43fc4c5ca4c746b659d61f5f"
format_stamp: "Formatted at 2026-08-06 11:46:01 on dist-test-slave-45dx"
server_key: "ef18484adc3300777489667158437e59"
server_key_iv: "16c690bbb6251c853dd1ad703267aca9"
server_key_version: "encryptionkey@0"
I20260806 11:46:01.682327 31528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "6c7b851e43fc4c5ca4c746b659d61f5f"
format_stamp: "Formatted at 2026-08-06 11:46:01 on dist-test-slave-45dx"
server_key: "ef18484adc3300777489667158437e59"
server_key_iv: "16c690bbb6251c853dd1ad703267aca9"
server_key_version: "encryptionkey@0"
I20260806 11:46:01.686056 31528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:46:01.688515 31555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:01.689682 31528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:46:01.689831 31528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "6c7b851e43fc4c5ca4c746b659d61f5f"
format_stamp: "Formatted at 2026-08-06 11:46:01 on dist-test-slave-45dx"
server_key: "ef18484adc3300777489667158437e59"
server_key_iv: "16c690bbb6251c853dd1ad703267aca9"
server_key_version: "encryptionkey@0"
I20260806 11:46:01.689955 31528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:01.714895 31528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:01.717881 31528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:01.718098 31528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:01.724977 31528 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41037
I20260806 11:46:01.725003 31640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41037 every 8 connection(s)
I20260806 11:46:01.725875 31528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:01.726975 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 31528
I20260806 11:46:01.727130 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:01.727433 452 external_mini_cluster.cc:1428] Setting key c5326260f6192a5d5ea34c5b72695473
I20260806 11:46:01.728834 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:01.734105 31645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f: Bootstrap starting.
Aug 06 11:46:01 dist-test-slave-45dx krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 11:46:01.736466 31645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:01.737172 31645 log.cc:824] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:01.739193 31645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f: No bootstrap required, opened a new log
I20260806 11:46:01.741484 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:01.728785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:57401 (local address 127.0.0.1:41037)
0806 11:46:01.729232 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:01.729247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:01.729282 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:46:01.729907 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:01.729925 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:01.730241 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:01.730569 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:01.731368 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:01.732406 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:01.733290 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:01.733752 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:01.733921 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:01.736727 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:01.736733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:01.736744 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:01.736770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:01.738598 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:01.739153 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:01.739157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:01.739161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:01.739278 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:01.739685 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:01.739688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:01.739689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:01.740047 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:01.740199 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:01.740553 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 11:46:01.740785 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":153,"threads_started":1}
I20260806 11:46:01.743284 31645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7b851e43fc4c5ca4c746b659d61f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } }
I20260806 11:46:01.743554 31645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:01.743611 31645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7b851e43fc4c5ca4c746b659d61f5f, State: Initialized, Role: FOLLOWER
I20260806 11:46:01.744065 31645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7b851e43fc4c5ca4c746b659d61f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } }
I20260806 11:46:01.744225 31645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:01.744297 31645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:01.744457 31645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:01.744607 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:01.745612 31645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7b851e43fc4c5ca4c746b659d61f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } }
I20260806 11:46:01.745977 31645 leader_election.cc:302] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c7b851e43fc4c5ca4c746b659d61f5f; no voters:
I20260806 11:46:01.746341 31645 leader_election.cc:288] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:01.746459 31656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:01.746712 31656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [term 1 LEADER]: Becoming Leader. State: Replica: 6c7b851e43fc4c5ca4c746b659d61f5f, State: Running, Role: LEADER
I20260806 11:46:01.747047 31656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7b851e43fc4c5ca4c746b659d61f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } }
I20260806 11:46:01.747643 31645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:01.748847 31657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c7b851e43fc4c5ca4c746b659d61f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7b851e43fc4c5ca4c746b659d61f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } } }
I20260806 11:46:01.748976 31657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:01.749182 31657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c7b851e43fc4c5ca4c746b659d61f5f. Latest consensus state: current_term: 1 leader_uuid: "6c7b851e43fc4c5ca4c746b659d61f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7b851e43fc4c5ca4c746b659d61f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } } }
I20260806 11:46:01.749249 31657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:01.750185 31667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:01.753623 31667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:01.756201 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:01.747141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:44329 (local address 127.0.0.1:41037)
0806 11:46:01.747249 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:01.747254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:01.747464 (+ 210us) server_negotiation.cc:407] Connection header received
0806 11:46:01.747552 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:01.747555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:01.747602 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:01.747689 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:01.748598 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:01.749418 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:01.750501 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:01.750717 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:01.750798 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:01.752396 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:01.752400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:01.752403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:01.752448 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:01.754157 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:01.754739 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:01.754744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:01.754746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:01.754840 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:01.755219 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:01.755222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:01.755224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:01.755451 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:01.755604 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:01.755940 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 11:46:01.756042 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 11:46:01.759682 31667 catalog_manager.cc:1347] Generated new cluster ID: 952ec484ed0446f4815ff5cd37f8108a
I20260806 11:46:01.759758 31667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:01.771940 31667 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:01.772893 31667 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:01.784004 31667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c7b851e43fc4c5ca4c746b659d61f5f: Generated new TSK 0
I20260806 11:46:01.784647 31667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:01.790357 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 31528
W20260806 11:46:01.809324 31509 connection.cc:535] client connection to 127.0.0.1:41037 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41037): Connection reset by peer (error 104)
2026-08-06T11:46:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1614 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:01 dist-test-slave-45dx krb5kdc[31684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:01 dist-test-slave-45dx krb5kdc[31684](info): set up 2 sockets
Aug 06 11:46:01 dist-test-slave-45dx krb5kdc[31684](info): commencing operation
krb5kdc: starting...
W20260806 11:46:03.016615 452 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:03 dist-test-slave-45dx krb5kdc[31684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:46:03.145602 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:03.261302 31703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:03.261667 31703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:03.261756 31703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:03.266011 31703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:03.266072 31703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:03.266091 31703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:03.266108 31703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:03.266168 31703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:03.271430 31703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:03.272539 31703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:03.273538 31703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:03.282603 31708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:03.282637 31709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:03.282603 31711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:03.282951 31703 server_base.cc:1034] running on GCE node
I20260806 11:46:03.283560 31703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:03.284766 31703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:03.285940 31703 hybrid_clock.cc:648] HybridClock initialized: now 1786016763285910 us; error 40 us; skew 500 ppm
Aug 06 11:46:03 dist-test-slave-45dx krb5kdc[31684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:03.288875 31703 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:46:03.289804 31703 webserver.cc:460] Webserver started at http://127.0.0.1:35759/ using document root <none> and password file <none>
I20260806 11:46:03.290374 31703 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:03.290474 31703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:03.290709 31703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:03.292840 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "c07e4bcf5dae450d97166f3e453e738d"
format_stamp: "Formatted at 2026-08-06 11:46:03 on dist-test-slave-45dx"
server_key: "d6da56c08f957b059370dd1f87540d22"
server_key_iv: "6d1f2ee573ceaaaa12170878d510d69d"
server_key_version: "encryptionkey@0"
I20260806 11:46:03.293430 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "c07e4bcf5dae450d97166f3e453e738d"
format_stamp: "Formatted at 2026-08-06 11:46:03 on dist-test-slave-45dx"
server_key: "d6da56c08f957b059370dd1f87540d22"
server_key_iv: "6d1f2ee573ceaaaa12170878d510d69d"
server_key_version: "encryptionkey@0"
I20260806 11:46:03.297178 31703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:46:03.299621 31724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:03.300892 31703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:46:03.301034 31703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "c07e4bcf5dae450d97166f3e453e738d"
format_stamp: "Formatted at 2026-08-06 11:46:03 on dist-test-slave-45dx"
server_key: "d6da56c08f957b059370dd1f87540d22"
server_key_iv: "6d1f2ee573ceaaaa12170878d510d69d"
server_key_version: "encryptionkey@0"
I20260806 11:46:03.301153 31703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:03.336073 31703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:03.339690 31703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:03.339913 31703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:03.347172 31703 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32833
I20260806 11:46:03.347203 31811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32833 every 8 connection(s)
I20260806 11:46:03.348148 31703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:03.350929 31812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:03.353315 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 31703
I20260806 11:46:03.353463 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:03.353785 452 external_mini_cluster.cc:1428] Setting key fcf07ceaa5bf512fb95af735ad7e2708
I20260806 11:46:03.357477 31812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d: Bootstrap starting.
I20260806 11:46:03.359997 31812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:03.360795 31812 log.cc:824] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:03.362633 31812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d: No bootstrap required, opened a new log
I20260806 11:46:03.363600 31815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:03.355241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:51223 (local address 127.0.0.1:32833)
0806 11:46:03.355682 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:03.355701 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:03.355742 (+ 41us) server_negotiation.cc:407] Connection header received
0806 11:46:03.356595 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:03.356618 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:03.357081 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:03.357425 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:03.357812 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:03.359272 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:03.360180 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:03.360732 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:03.360919 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:03.361881 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:03.361887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:03.361909 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:03.362287 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:03.362348 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:03.362372 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:03.362514 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:03.362689 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 11:46:03.362833 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260806 11:46:03.366473 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:03.366472 31812 raft_consensus.cc:348] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07e4bcf5dae450d97166f3e453e738d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } }
I20260806 11:46:03.366750 31812 raft_consensus.cc:374] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:03.366828 31812 raft_consensus.cc:729] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07e4bcf5dae450d97166f3e453e738d, State: Initialized, Role: FOLLOWER
I20260806 11:46:03.367218 31812 consensus_queue.cc:260] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07e4bcf5dae450d97166f3e453e738d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } }
I20260806 11:46:03.367367 31812 raft_consensus.cc:388] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:03.367460 31812 raft_consensus.cc:482] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:03.367563 31812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:03.368592 31812 raft_consensus.cc:504] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07e4bcf5dae450d97166f3e453e738d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } }
I20260806 11:46:03.368925 31812 leader_election.cc:302] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c07e4bcf5dae450d97166f3e453e738d; no voters:
I20260806 11:46:03.369266 31812 leader_election.cc:288] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:03.369364 31824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:03.369660 31824 raft_consensus.cc:686] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [term 1 LEADER]: Becoming Leader. State: Replica: c07e4bcf5dae450d97166f3e453e738d, State: Running, Role: LEADER
I20260806 11:46:03.370062 31824 consensus_queue.cc:237] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07e4bcf5dae450d97166f3e453e738d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } }
I20260806 11:46:03.370561 31812 sys_catalog.cc:564] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:03.372761 31829 sys_catalog.cc:455] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c07e4bcf5dae450d97166f3e453e738d. Latest consensus state: current_term: 1 leader_uuid: "c07e4bcf5dae450d97166f3e453e738d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07e4bcf5dae450d97166f3e453e738d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } } }
I20260806 11:46:03.372936 31829 sys_catalog.cc:458] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:03.372757 31825 sys_catalog.cc:455] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c07e4bcf5dae450d97166f3e453e738d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07e4bcf5dae450d97166f3e453e738d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } } }
I20260806 11:46:03.373687 31825 sys_catalog.cc:458] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:03.374173 31815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:03.368830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:57771 (local address 127.0.0.1:32833)
0806 11:46:03.368944 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:03.368948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:03.369338 (+ 390us) server_negotiation.cc:407] Connection header received
0806 11:46:03.369497 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:03.369500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:03.369552 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:03.369640 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:03.370050 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:03.370800 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:03.372012 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:03.372355 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:03.372460 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:03.373587 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:03.373591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:03.373594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:03.373608 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:03.373639 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:03.373656 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:03.373725 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:03.374012 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 11:46:03.374052 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 11:46:03.375516 31846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:03.375605 31846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:03.375706 31848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:03.376580 31848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:03.382109 31848 catalog_manager.cc:1347] Generated new cluster ID: ce5ba531767e4b4095d364e9c42c7cc8
I20260806 11:46:03.382189 31848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:03.389982 31848 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:03.390934 31848 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:03.399153 31848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c07e4bcf5dae450d97166f3e453e738d: Generated new TSK 0
I20260806 11:46:03.399839 31848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:03.404403 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 31703
W20260806 11:46:03.422391 31682 connection.cc:535] client connection to 127.0.0.1:32833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32833): Connection reset by peer (error 104)
2026-08-06T11:46:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1612 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:03 dist-test-slave-45dx krb5kdc[31855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:03 dist-test-slave-45dx krb5kdc[31855](info): set up 2 sockets
Aug 06 11:46:03 dist-test-slave-45dx krb5kdc[31855](info): commencing operation
krb5kdc: starting...
W20260806 11:46:04.624493 452 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:04 dist-test-slave-45dx krb5kdc[31855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 11:46:04.753647 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:04.864465 31879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:04.864711 31879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:04.864751 31879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:04.868729 31879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:04.868785 31879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:04.868804 31879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:04.868821 31879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:04.868836 31879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:04.873826 31879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:04.874761 31879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:04.875679 31879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:04.883809 31889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:04.883821 31886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:04.883971 31879 server_base.cc:1034] running on GCE node
W20260806 11:46:04.883824 31884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:04.884694 31879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:04.885665 31879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:04.886837 31879 hybrid_clock.cc:648] HybridClock initialized: now 1786016764886819 us; error 29 us; skew 500 ppm
Aug 06 11:46:04 dist-test-slave-45dx krb5kdc[31855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:04.889541 31879 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 11:46:04.890372 31879 webserver.cc:460] Webserver started at http://127.0.0.1:39215/ using document root <none> and password file <none>
I20260806 11:46:04.890895 31879 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:04.890951 31879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:04.891108 31879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:04.893062 31879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "32bb97df15db4d0fb9dd3bbdf140b688"
format_stamp: "Formatted at 2026-08-06 11:46:04 on dist-test-slave-45dx"
server_key: "3f5c4b1702b205c0f4080c9b7f37762f"
server_key_iv: "fdca7329f6230c68d4b4e9cc9817be6b"
server_key_version: "encryptionkey@0"
I20260806 11:46:04.893553 31879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "32bb97df15db4d0fb9dd3bbdf140b688"
format_stamp: "Formatted at 2026-08-06 11:46:04 on dist-test-slave-45dx"
server_key: "3f5c4b1702b205c0f4080c9b7f37762f"
server_key_iv: "fdca7329f6230c68d4b4e9cc9817be6b"
server_key_version: "encryptionkey@0"
I20260806 11:46:04.896947 31879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 11:46:04.899206 31898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:04.900414 31879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:46:04.900553 31879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "32bb97df15db4d0fb9dd3bbdf140b688"
format_stamp: "Formatted at 2026-08-06 11:46:04 on dist-test-slave-45dx"
server_key: "3f5c4b1702b205c0f4080c9b7f37762f"
server_key_iv: "fdca7329f6230c68d4b4e9cc9817be6b"
server_key_version: "encryptionkey@0"
I20260806 11:46:04.900651 31879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:04.904875 31879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:04.907931 31879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:04.908083 31879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:04.916102 31879 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44449
I20260806 11:46:04.916132 31975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44449 every 8 connection(s)
I20260806 11:46:04.917352 31879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:04.919919 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 31879
I20260806 11:46:04.920084 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:04.920539 452 external_mini_cluster.cc:1428] Setting key 1576613d28982feade2226b1551d5c05
I20260806 11:46:04.921119 31976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:04.925812 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:04.921980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:60361 (local address 127.0.0.1:44449)
0806 11:46:04.922413 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:04.922432 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:04.922659 (+ 227us) server_negotiation.cc:407] Connection header received
0806 11:46:04.923384 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:04.923399 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:04.923644 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:04.923890 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:04.924710 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:04.924738 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:04.924750 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:04.924760 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:04.924816 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:04.924834 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:04.924942 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:04.925062 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 11:46:04.925177 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
I20260806 11:46:04.928167 31976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688: Bootstrap starting.
I20260806 11:46:04.928489 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:04.930353 31976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:04.931285 31976 log.cc:824] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:04.932186 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:04.930475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:37545 (local address 127.0.0.1:44449)
0806 11:46:04.930583 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:04.930586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:04.930886 (+ 300us) server_negotiation.cc:407] Connection header received
0806 11:46:04.931005 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:04.931006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:04.931032 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:04.931092 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:04.931812 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:04.931815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:04.931817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:04.931828 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:04.931852 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:04.931863 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:04.931966 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:04.932040 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 11:46:04.932077 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 11:46:04.933341 31976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688: No bootstrap required, opened a new log
I20260806 11:46:04.936642 31976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bb97df15db4d0fb9dd3bbdf140b688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44449 } }
I20260806 11:46:04.936900 31976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:04.936975 31976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bb97df15db4d0fb9dd3bbdf140b688, State: Initialized, Role: FOLLOWER
I20260806 11:46:04.937359 31976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bb97df15db4d0fb9dd3bbdf140b688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44449 } }
I20260806 11:46:04.937511 31976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:04.937597 31976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:04.937690 31976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:04.938463 31976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bb97df15db4d0fb9dd3bbdf140b688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44449 } }
I20260806 11:46:04.938786 31976 leader_election.cc:302] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32bb97df15db4d0fb9dd3bbdf140b688; no voters:
I20260806 11:46:04.939071 31976 leader_election.cc:288] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:04.939175 31988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:04.939409 31988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [term 1 LEADER]: Becoming Leader. State: Replica: 32bb97df15db4d0fb9dd3bbdf140b688, State: Running, Role: LEADER
I20260806 11:46:04.939743 31988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bb97df15db4d0fb9dd3bbdf140b688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44449 } }
I20260806 11:46:04.940241 31976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:04.941334 31989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32bb97df15db4d0fb9dd3bbdf140b688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bb97df15db4d0fb9dd3bbdf140b688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44449 } } }
I20260806 11:46:04.941471 31989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:04.941860 31998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:04.941831 31990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32bb97df15db4d0fb9dd3bbdf140b688. Latest consensus state: current_term: 1 leader_uuid: "32bb97df15db4d0fb9dd3bbdf140b688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bb97df15db4d0fb9dd3bbdf140b688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44449 } } }
I20260806 11:46:04.942821 31990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:04.944482 31998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:04.949127 31998 catalog_manager.cc:1347] Generated new cluster ID: c8c0027e81744ca180b4365c690a0215
I20260806 11:46:04.949188 31998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:04.967767 31998 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:04.968091 31998 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:04.976490 31998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32bb97df15db4d0fb9dd3bbdf140b688: Generated new TSK 0
I20260806 11:46:04.977245 31998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:04.987787 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 31879
2026-08-06T11:46:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1578 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:05 dist-test-slave-45dx krb5kdc[32013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:05 dist-test-slave-45dx krb5kdc[32013](info): set up 2 sockets
Aug 06 11:46:05 dist-test-slave-45dx krb5kdc[32013](info): commencing operation
krb5kdc: starting...
W20260806 11:46:06.233461 452 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:06 dist-test-slave-45dx krb5kdc[32013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:46:06.371040 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:41501
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:41501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:06.484200 32028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:06.484669 32028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:06.484756 32028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:06.489169 32028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:06.489264 32028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:06.489301 32028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:06.489341 32028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:06.489374 32028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:06.494589 32028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:41501
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:41501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:06.495676 32028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:06.496755 32028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:06.505391 32037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:06.505415 32028 server_base.cc:1034] running on GCE node
W20260806 11:46:06.505342 32034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:06.505357 32047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:06.506251 32028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:06.507275 32028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:06.508441 32028 hybrid_clock.cc:648] HybridClock initialized: now 1786016766508421 us; error 32 us; skew 500 ppm
Aug 06 11:46:06 dist-test-slave-45dx krb5kdc[32013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:06.511173 32028 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:46:06.512012 32028 webserver.cc:460] Webserver started at http://127.0.113.62:39179/ using document root <none> and password file <none>
I20260806 11:46:06.512624 32028 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:06.512692 32028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:06.512861 32028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:06.514861 32028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "bbbf50c67cac4e42b2220df142d71df0"
format_stamp: "Formatted at 2026-08-06 11:46:06 on dist-test-slave-45dx"
server_key: "72a90c1d827c2a4815eda0ac7f698665"
server_key_iv: "bb686c6b4bcb1045b597522c548a4a26"
server_key_version: "encryptionkey@0"
I20260806 11:46:06.515483 32028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "bbbf50c67cac4e42b2220df142d71df0"
format_stamp: "Formatted at 2026-08-06 11:46:06 on dist-test-slave-45dx"
server_key: "72a90c1d827c2a4815eda0ac7f698665"
server_key_iv: "bb686c6b4bcb1045b597522c548a4a26"
server_key_version: "encryptionkey@0"
I20260806 11:46:06.519375 32028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 11:46:06.521754 32057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:06.523047 32028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:46:06.523196 32028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "bbbf50c67cac4e42b2220df142d71df0"
format_stamp: "Formatted at 2026-08-06 11:46:06 on dist-test-slave-45dx"
server_key: "72a90c1d827c2a4815eda0ac7f698665"
server_key_iv: "bb686c6b4bcb1045b597522c548a4a26"
server_key_version: "encryptionkey@0"
I20260806 11:46:06.523331 32028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:06.539265 32028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:06.542644 32028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:06.542861 32028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:06.550247 32028 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:41501
I20260806 11:46:06.550379 32130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:41501 every 8 connection(s)
I20260806 11:46:06.551263 32028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:06.554047 32136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:06.558100 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 32028
I20260806 11:46:06.558254 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:06.558552 452 external_mini_cluster.cc:1428] Setting key 58832637a85600623fc78a865543ac4f
I20260806 11:46:06.559895 32136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0: Bootstrap starting.
I20260806 11:46:06.562657 32136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:06.563657 32136 log.cc:824] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:06.565778 32136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0: No bootstrap required, opened a new log
Aug 06 11:46:06 dist-test-slave-45dx krb5kdc[32013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:06.569936 32136 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf50c67cac4e42b2220df142d71df0" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41501 } }
I20260806 11:46:06.570227 32136 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:06.570286 32136 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbf50c67cac4e42b2220df142d71df0, State: Initialized, Role: FOLLOWER
I20260806 11:46:06.570792 32136 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf50c67cac4e42b2220df142d71df0" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41501 } }
I20260806 11:46:06.571012 32136 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:06.571087 32136 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:06.571195 32136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:06.572434 32136 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf50c67cac4e42b2220df142d71df0" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41501 } }
I20260806 11:46:06.572819 32136 leader_election.cc:302] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbbf50c67cac4e42b2220df142d71df0; no voters:
I20260806 11:46:06.573210 32136 leader_election.cc:288] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:06.573585 32145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:06.573864 32145 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [term 1 LEADER]: Becoming Leader. State: Replica: bbbf50c67cac4e42b2220df142d71df0, State: Running, Role: LEADER
I20260806 11:46:06.574350 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:06.560100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50289 (local address 127.0.113.62:41501)
0806 11:46:06.560528 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:06.560547 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:06.560590 (+ 43us) server_negotiation.cc:407] Connection header received
0806 11:46:06.561456 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:06.561479 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:06.561913 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:06.562271 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:06.563008 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:06.564457 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:06.565321 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:06.566001 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:06.566174 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:06.568742 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:06.568749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:06.568766 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:06.568798 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:06.570776 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:06.571410 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:06.571419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:06.571428 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:06.571578 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:06.571918 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:06.571922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:06.571924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:06.572531 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:06.572757 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:06.572995 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 11:46:06.573241 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":149,"threads_started":1}
I20260806 11:46:06.574306 32145 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf50c67cac4e42b2220df142d71df0" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41501 } }
I20260806 11:46:06.574539 32136 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:06.577571 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:06.577230 32147 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbbf50c67cac4e42b2220df142d71df0. Latest consensus state: current_term: 1 leader_uuid: "bbbf50c67cac4e42b2220df142d71df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf50c67cac4e42b2220df142d71df0" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41501 } } }
I20260806 11:46:06.579658 32147 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:06.577586 32144 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbbf50c67cac4e42b2220df142d71df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf50c67cac4e42b2220df142d71df0" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 41501 } } }
I20260806 11:46:06.579949 32144 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0 [sys.catalog]: This master's current role is: LEADER
W20260806 11:46:06.582036 32165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:06.582118 32165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:06.582185 32173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:06.582880 32173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:06.588770 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:06.579763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42633 (local address 127.0.113.62:41501)
0806 11:46:06.579880 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:06.579886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:06.580385 (+ 499us) server_negotiation.cc:407] Connection header received
0806 11:46:06.580536 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:06.580539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:06.580590 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:06.580676 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:06.581370 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:06.581936 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:06.582830 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:06.583131 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:06.583241 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:06.584962 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:06.584965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:06.584968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:06.584994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:06.586754 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:06.587381 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:06.587384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:06.587386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:06.587474 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:06.587822 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:06.587826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:06.587828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:06.588017 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:06.588178 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:06.588502 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 11:46:06.588601 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 11:46:06.589025 32173 catalog_manager.cc:1347] Generated new cluster ID: f0e6241b0a964f888d662cdabe55f5b6
I20260806 11:46:06.589090 32173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:06.622304 32173 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:06.623358 32173 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:06.632171 32173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbbf50c67cac4e42b2220df142d71df0: Generated new TSK 0
I20260806 11:46:06.633134 32173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:06.648904 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 32028
W20260806 11:46:06.662863 32011 connection.cc:535] client connection to 127.0.113.62:41501 recv error: Network error: failed to read from TLS socket (remote: 127.0.113.62:41501): Connection reset by peer (error 104)
2026-08-06T11:46:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1661 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:06 dist-test-slave-45dx krb5kdc[32182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:06 dist-test-slave-45dx krb5kdc[32182](info): set up 2 sockets
Aug 06 11:46:06 dist-test-slave-45dx krb5kdc[32182](info): commencing operation
krb5kdc: starting...
W20260806 11:46:07.872503 452 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:07 dist-test-slave-45dx krb5kdc[32182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:46:07.999679 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:39547
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:39547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:08.113449 32202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:08.113749 32202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:08.113830 32202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:08.117916 32202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:08.118000 32202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:08.118038 32202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:08.118074 32202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:08.118108 32202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:08.123538 32202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:39547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:39547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:08.124692 32202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:08.125711 32202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:08.134421 32210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:08.134481 32209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:08.134445 32214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:08.134618 32202 server_base.cc:1034] running on GCE node
I20260806 11:46:08.135342 32202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:08.136559 32202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:08.137750 32202 hybrid_clock.cc:648] HybridClock initialized: now 1786016768137733 us; error 31 us; skew 500 ppm
Aug 06 11:46:08 dist-test-slave-45dx krb5kdc[32182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:08.144405 32202 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:46:08.145349 32202 webserver.cc:460] Webserver started at http://127.0.113.62:34131/ using document root <none> and password file <none>
I20260806 11:46:08.145901 32202 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:08.145982 32202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:08.146157 32202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:08.148290 32202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "7fd33b7aba93470f86c26501590e6535"
format_stamp: "Formatted at 2026-08-06 11:46:08 on dist-test-slave-45dx"
server_key: "87b5f4c32ab72414cb08bd950afdc1bf"
server_key_iv: "edd8cfab3a16bfebc85086de29cdf7ea"
server_key_version: "encryptionkey@0"
I20260806 11:46:08.148933 32202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "7fd33b7aba93470f86c26501590e6535"
format_stamp: "Formatted at 2026-08-06 11:46:08 on dist-test-slave-45dx"
server_key: "87b5f4c32ab72414cb08bd950afdc1bf"
server_key_iv: "edd8cfab3a16bfebc85086de29cdf7ea"
server_key_version: "encryptionkey@0"
I20260806 11:46:08.152769 32202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 11:46:08.155210 32222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:08.156493 32202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:46:08.156598 32202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "7fd33b7aba93470f86c26501590e6535"
format_stamp: "Formatted at 2026-08-06 11:46:08 on dist-test-slave-45dx"
server_key: "87b5f4c32ab72414cb08bd950afdc1bf"
server_key_iv: "edd8cfab3a16bfebc85086de29cdf7ea"
server_key_version: "encryptionkey@0"
I20260806 11:46:08.156685 32202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:08.181565 32202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:08.184731 32202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:08.184907 32202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:08.192246 32202 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:39547
I20260806 11:46:08.192263 32301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:39547 every 8 connection(s)
I20260806 11:46:08.193320 32202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:08.196566 32303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:08.197636 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 32202
I20260806 11:46:08.197815 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:08.198112 452 external_mini_cluster.cc:1428] Setting key ad9fdee9009d0e3ee12297bf20d7eb95
I20260806 11:46:08.204007 32303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535: Bootstrap starting.
I20260806 11:46:08.206876 32303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:08.207906 32303 log.cc:824] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:08.209285 32308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:08.199621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36435 (local address 127.0.113.62:39547)
0806 11:46:08.200085 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:08.200106 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:08.200150 (+ 44us) server_negotiation.cc:407] Connection header received
0806 11:46:08.201046 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:08.201070 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:08.201664 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:08.202132 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:08.202599 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:08.204233 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:08.205187 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:08.205965 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:08.206138 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:08.207073 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:08.207080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:08.207098 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:08.207658 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:08.207737 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:08.207776 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:08.207935 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:08.208113 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 11:46:08.208257 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":132,"threads_started":1}
I20260806 11:46:08.210268 32303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535: No bootstrap required, opened a new log
I20260806 11:46:08.212631 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:08.215580 32303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd33b7aba93470f86c26501590e6535" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 39547 } }
I20260806 11:46:08.215947 32303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:08.216023 32303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd33b7aba93470f86c26501590e6535, State: Initialized, Role: FOLLOWER
I20260806 11:46:08.216562 32303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd33b7aba93470f86c26501590e6535" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 39547 } }
I20260806 11:46:08.216738 32303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:08.216845 32303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:08.216969 32303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:08.217995 32303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd33b7aba93470f86c26501590e6535" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 39547 } }
I20260806 11:46:08.218372 32303 leader_election.cc:302] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fd33b7aba93470f86c26501590e6535; no voters:
I20260806 11:46:08.218796 32303 leader_election.cc:288] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:08.218914 32317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:08.219130 32317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd33b7aba93470f86c26501590e6535, State: Running, Role: LEADER
I20260806 11:46:08.219509 32317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd33b7aba93470f86c26501590e6535" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 39547 } }
I20260806 11:46:08.219657 32308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:08.215245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57089 (local address 127.0.113.62:39547)
0806 11:46:08.215388 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:08.215394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:08.215574 (+ 180us) server_negotiation.cc:407] Connection header received
0806 11:46:08.215712 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:08.215714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:08.215754 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:08.215818 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:08.216170 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:08.216756 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:08.217681 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:08.217924 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:08.218037 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:08.219089 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:08.219094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:08.219097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:08.219111 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:08.219143 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:08.219158 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:08.219249 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:08.219475 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 11:46:08.219514 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:46:08.220404 32303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:08.220794 32317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fd33b7aba93470f86c26501590e6535. Latest consensus state: current_term: 1 leader_uuid: "7fd33b7aba93470f86c26501590e6535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd33b7aba93470f86c26501590e6535" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 39547 } } }
I20260806 11:46:08.221206 32317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:08.221274 32318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fd33b7aba93470f86c26501590e6535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd33b7aba93470f86c26501590e6535" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 39547 } } }
I20260806 11:46:08.221392 32318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:08.221714 32324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:08.224921 32324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:08.231178 32324 catalog_manager.cc:1347] Generated new cluster ID: 260e502511ff4eb8b62da73ddd9c08aa
I20260806 11:46:08.231259 32324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:08.243005 32324 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:08.244017 32324 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:08.252251 32324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fd33b7aba93470f86c26501590e6535: Generated new TSK 0
I20260806 11:46:08.253146 32324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:08.262940 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 32202
W20260806 11:46:08.276875 32180 connection.cc:535] client connection to 127.0.113.62:39547 recv error: Network error: failed to read from TLS socket (remote: 127.0.113.62:39547): Connection reset by peer (error 104)
2026-08-06T11:46:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:08 dist-test-slave-45dx krb5kdc[32347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:08 dist-test-slave-45dx krb5kdc[32347](info): set up 2 sockets
Aug 06 11:46:08 dist-test-slave-45dx krb5kdc[32347](info): commencing operation
krb5kdc: starting...
W20260806 11:46:09.494280 452 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:09 dist-test-slave-45dx krb5kdc[32347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:46:09.627753 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:43377
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:43377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:09.740602 32369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:09.740988 32369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:09.741074 32369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:09.745158 32369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:09.745273 32369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:09.745311 32369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:09.745349 32369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:09.745383 32369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:09.750948 32369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:43377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:09.752192 32369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:09.753350 32369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:09.762506 32376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:09.762507 32377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:09.762507 32380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:09.762867 32369 server_base.cc:1034] running on GCE node
I20260806 11:46:09.763602 32369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:09.764919 32369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:09.766119 32369 hybrid_clock.cc:648] HybridClock initialized: now 1786016769766081 us; error 52 us; skew 500 ppm
Aug 06 11:46:09 dist-test-slave-45dx krb5kdc[32347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:09.768975 32369 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:46:09.769884 32369 webserver.cc:460] Webserver started at http://127.0.113.62:40311/ using document root <none> and password file <none>
I20260806 11:46:09.770484 32369 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:09.770581 32369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:09.770828 32369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:09.773031 32369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "fb94124447204bfbae28ed0e3b7a1806"
format_stamp: "Formatted at 2026-08-06 11:46:09 on dist-test-slave-45dx"
server_key: "9cae208d22a2d90938e1387172051f25"
server_key_iv: "11903bb18c23c65a33358597dfec7ab3"
server_key_version: "encryptionkey@0"
I20260806 11:46:09.773634 32369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "fb94124447204bfbae28ed0e3b7a1806"
format_stamp: "Formatted at 2026-08-06 11:46:09 on dist-test-slave-45dx"
server_key: "9cae208d22a2d90938e1387172051f25"
server_key_iv: "11903bb18c23c65a33358597dfec7ab3"
server_key_version: "encryptionkey@0"
I20260806 11:46:09.777470 32369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 11:46:09.779927 32389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:09.781281 32369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:46:09.781391 32369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "fb94124447204bfbae28ed0e3b7a1806"
format_stamp: "Formatted at 2026-08-06 11:46:09 on dist-test-slave-45dx"
server_key: "9cae208d22a2d90938e1387172051f25"
server_key_iv: "11903bb18c23c65a33358597dfec7ab3"
server_key_version: "encryptionkey@0"
I20260806 11:46:09.781494 32369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:09.786043 32369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:09.789069 32369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:09.789223 32369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:09.796192 32369 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:43377
I20260806 11:46:09.796252 32477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:43377 every 8 connection(s)
I20260806 11:46:09.797209 32369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:09.799949 32478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:09.804818 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 32369
I20260806 11:46:09.805007 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:09.805280 452 external_mini_cluster.cc:1428] Setting key b6840aa70888f32312cb125b582f350f
I20260806 11:46:09.805671 32478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806: Bootstrap starting.
I20260806 11:46:09.808318 32478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:09.809237 32478 log.cc:824] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:09.810861 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:09.806746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38267 (local address 127.0.113.62:43377)
0806 11:46:09.807177 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:09.807196 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:09.807237 (+ 41us) server_negotiation.cc:407] Connection header received
0806 11:46:09.808039 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:09.808062 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:09.808455 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:09.808829 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:09.809642 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:09.809649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:09.809666 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:09.809679 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:09.809743 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:09.809764 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:09.809897 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:09.810028 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 11:46:09.810180 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":117,"threads_started":1}
I20260806 11:46:09.811365 32478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806: No bootstrap required, opened a new log
I20260806 11:46:09.813884 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:09.815810 32478 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb94124447204bfbae28ed0e3b7a1806" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43377 } }
I20260806 11:46:09.816138 32478 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:09.816218 32478 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb94124447204bfbae28ed0e3b7a1806, State: Initialized, Role: FOLLOWER
I20260806 11:46:09.816825 32478 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb94124447204bfbae28ed0e3b7a1806" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43377 } }
I20260806 11:46:09.817016 32478 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:09.817092 32478 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:09.817201 32478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:09.817812 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:09.815962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43445 (local address 127.0.113.62:43377)
0806 11:46:09.816091 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:09.816102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:09.816251 (+ 149us) server_negotiation.cc:407] Connection header received
0806 11:46:09.816415 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:09.816417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:09.816450 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:09.816518 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:09.817389 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:09.817395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:09.817402 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:09.817418 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:09.817450 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:09.817478 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:09.817565 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:09.817652 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 11:46:09.817690 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 11:46:09.818506 32478 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb94124447204bfbae28ed0e3b7a1806" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43377 } }
I20260806 11:46:09.818912 32478 leader_election.cc:302] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb94124447204bfbae28ed0e3b7a1806; no voters:
I20260806 11:46:09.819286 32478 leader_election.cc:288] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:09.819375 32492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:09.819581 32492 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [term 1 LEADER]: Becoming Leader. State: Replica: fb94124447204bfbae28ed0e3b7a1806, State: Running, Role: LEADER
I20260806 11:46:09.819892 32492 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb94124447204bfbae28ed0e3b7a1806" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43377 } }
I20260806 11:46:09.820672 32478 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:09.821717 32494 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb94124447204bfbae28ed0e3b7a1806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb94124447204bfbae28ed0e3b7a1806" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43377 } } }
I20260806 11:46:09.821830 32493 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb94124447204bfbae28ed0e3b7a1806. Latest consensus state: current_term: 1 leader_uuid: "fb94124447204bfbae28ed0e3b7a1806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb94124447204bfbae28ed0e3b7a1806" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43377 } } }
I20260806 11:46:09.822039 32494 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:09.822053 32493 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:09.822983 32503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:09.826292 32503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:09.832046 32503 catalog_manager.cc:1347] Generated new cluster ID: c3daf2db470946018caa09fe60d5b7a1
I20260806 11:46:09.832154 32503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:09.841897 32503 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:09.842228 32503 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:09.854364 32503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb94124447204bfbae28ed0e3b7a1806: Generated new TSK 0
I20260806 11:46:09.855249 32503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:09.866206 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 32369
2026-08-06T11:46:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1600 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:09 dist-test-slave-45dx krb5kdc[32520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:09 dist-test-slave-45dx krb5kdc[32520](info): set up 2 sockets
Aug 06 11:46:09 dist-test-slave-45dx krb5kdc[32520](info): commencing operation
krb5kdc: starting...
W20260806 11:46:11.093223 452 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:11 dist-test-slave-45dx krb5kdc[32520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:46:11.227933 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:44405
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:11.344007 32543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:11.344391 32543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:11.344494 32543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:11.348621 32543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:11.348711 32543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:11.348750 32543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:11.348802 32543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:11.348837 32543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:11.354193 32543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:11.355280 32543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:11.356361 32543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:11.364989 32554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:11.365026 32551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:11.365031 32552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:11.365224 32543 server_base.cc:1034] running on GCE node
I20260806 11:46:11.365935 32543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:11.367046 32543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:11.368216 32543 hybrid_clock.cc:648] HybridClock initialized: now 1786016771368188 us; error 38 us; skew 500 ppm
Aug 06 11:46:11 dist-test-slave-45dx krb5kdc[32520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:11.371397 32543 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:46:11.372290 32543 webserver.cc:460] Webserver started at http://127.0.113.62:46331/ using document root <none> and password file <none>
I20260806 11:46:11.372912 32543 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:11.372996 32543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:11.373220 32543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:11.375352 32543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "721740398af24aaf953a6f031a63bf45"
format_stamp: "Formatted at 2026-08-06 11:46:11 on dist-test-slave-45dx"
server_key: "4d63ce4bcf513fc3423b7efe2549f136"
server_key_iv: "92fff391a6068edcb3faf978cb2fe17b"
server_key_version: "encryptionkey@0"
I20260806 11:46:11.375952 32543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "721740398af24aaf953a6f031a63bf45"
format_stamp: "Formatted at 2026-08-06 11:46:11 on dist-test-slave-45dx"
server_key: "4d63ce4bcf513fc3423b7efe2549f136"
server_key_iv: "92fff391a6068edcb3faf978cb2fe17b"
server_key_version: "encryptionkey@0"
I20260806 11:46:11.379863 32543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260806 11:46:11.382452 32568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:11.383715 32543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:46:11.383864 32543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "721740398af24aaf953a6f031a63bf45"
format_stamp: "Formatted at 2026-08-06 11:46:11 on dist-test-slave-45dx"
server_key: "4d63ce4bcf513fc3423b7efe2549f136"
server_key_iv: "92fff391a6068edcb3faf978cb2fe17b"
server_key_version: "encryptionkey@0"
I20260806 11:46:11.383986 32543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:11.407217 32543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:11.410401 32543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:11.410631 32543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:11.417836 32543 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:44405
I20260806 11:46:11.417856 32641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:44405 every 8 connection(s)
I20260806 11:46:11.418828 32543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:11.421696 32642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:11.426291 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 32543
I20260806 11:46:11.426452 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:11.426757 452 external_mini_cluster.cc:1428] Setting key 6749e461e57b15e9681154d40f63db1c
I20260806 11:46:11.427822 32642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45: Bootstrap starting.
I20260806 11:46:11.430315 32642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:11.431047 32642 log.cc:824] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:11.433029 32642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45: No bootstrap required, opened a new log
Aug 06 11:46:11 dist-test-slave-45dx krb5kdc[32520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:11.436884 32642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721740398af24aaf953a6f031a63bf45" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44405 } }
I20260806 11:46:11.437196 32642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:11.437268 32642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721740398af24aaf953a6f031a63bf45, State: Initialized, Role: FOLLOWER
I20260806 11:46:11.437813 32642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721740398af24aaf953a6f031a63bf45" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44405 } }
I20260806 11:46:11.437984 32642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:11.438056 32642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:11.438153 32642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:11.439323 32642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721740398af24aaf953a6f031a63bf45" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44405 } }
I20260806 11:46:11.439677 32642 leader_election.cc:302] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 721740398af24aaf953a6f031a63bf45; no voters:
I20260806 11:46:11.440006 32642 leader_election.cc:288] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:11.440099 32647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:11.440418 32647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [term 1 LEADER]: Becoming Leader. State: Replica: 721740398af24aaf953a6f031a63bf45, State: Running, Role: LEADER
I20260806 11:46:11.440796 32647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721740398af24aaf953a6f031a63bf45" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44405 } }
I20260806 11:46:11.441387 32642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:11.442009 32647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 721740398af24aaf953a6f031a63bf45. Latest consensus state: current_term: 1 leader_uuid: "721740398af24aaf953a6f031a63bf45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721740398af24aaf953a6f031a63bf45" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44405 } } }
I20260806 11:46:11.442142 32647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:11.442265 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:11.428322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53771 (local address 127.0.113.62:44405)
0806 11:46:11.428757 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:11.428773 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:11.428826 (+ 53us) server_negotiation.cc:407] Connection header received
0806 11:46:11.429544 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:11.429561 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:11.429908 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:11.430260 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:11.431103 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:11.432350 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:11.433221 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:11.433884 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:11.434075 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:11.436471 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:11.436477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:11.436493 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:11.436525 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:11.438628 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:11.439060 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:11.439067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:11.439072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:11.439236 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:11.439535 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:11.439538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:11.439540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:11.440031 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:11.440329 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:11.440955 (+ 626us) server_negotiation.cc:299] Negotiation successful
0806 11:46:11.441265 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":138,"threads_started":1}
I20260806 11:46:11.442582 32651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "721740398af24aaf953a6f031a63bf45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721740398af24aaf953a6f031a63bf45" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 44405 } } }
I20260806 11:46:11.442807 32651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:11.445595 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 11:46:11.446450 32666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:11.446528 32666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:11.446607 32655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:11.447525 32655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:11.453936 32655 catalog_manager.cc:1347] Generated new cluster ID: bebda400041747f7a1fdc067393c8bc9
I20260806 11:46:11.454020 32655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:11.456598 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:11.447538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56183 (local address 127.0.113.62:44405)
0806 11:46:11.447636 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:11.447640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:11.448690 (+ 1050us) server_negotiation.cc:407] Connection header received
0806 11:46:11.448831 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:11.448835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:11.448886 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:11.448967 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:11.449745 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:11.450271 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:11.451072 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:11.451308 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:11.451417 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:11.453140 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:11.453145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:11.453151 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:11.453177 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:11.454859 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:11.455325 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:11.455329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:11.455331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:11.455423 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:11.455704 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:11.455707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:11.455709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:11.455943 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:11.456084 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:11.456319 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 11:46:11.456446 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 11:46:11.461946 32655 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:11.463006 32655 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:11.469655 32655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 721740398af24aaf953a6f031a63bf45: Generated new TSK 0
I20260806 11:46:11.470299 32655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:11.477033 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 32543
2026-08-06T11:46:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1616 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:11 dist-test-slave-45dx krb5kdc[32682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:11 dist-test-slave-45dx krb5kdc[32682](info): set up 2 sockets
Aug 06 11:46:11 dist-test-slave-45dx krb5kdc[32682](info): commencing operation
krb5kdc: starting...
W20260806 11:46:12.714629 452 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:12 dist-test-slave-45dx krb5kdc[32682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:46:12.845717 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33493
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:33493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:12.956751 32702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:12.957100 32702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:12.957193 32702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:12.961324 32702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:12.961405 32702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:12.961444 32702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:12.961498 32702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:12.961539 32702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:12.966766 32702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:33493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:12.967852 32702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:12.968950 32702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:12.977494 32710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:12.977463 32711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:12.977631 32702 server_base.cc:1034] running on GCE node
W20260806 11:46:12.977471 32714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:12.978360 32702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:12.979420 32702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:12.980608 32702 hybrid_clock.cc:648] HybridClock initialized: now 1786016772980581 us; error 41 us; skew 500 ppm
Aug 06 11:46:12 dist-test-slave-45dx krb5kdc[32682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:12.983318 32702 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:46:12.984186 32702 webserver.cc:460] Webserver started at http://127.0.113.62:37849/ using document root <none> and password file <none>
I20260806 11:46:12.984789 32702 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:12.984881 32702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:12.985108 32702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:12.987134 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "d4a4b9786fa0406fa70321ebc813e05e"
format_stamp: "Formatted at 2026-08-06 11:46:12 on dist-test-slave-45dx"
server_key: "1de8e5ceee3345d06a8913e066eef344"
server_key_iv: "e247a4b93d13f86261eb4d1e986d0bbc"
server_key_version: "encryptionkey@0"
I20260806 11:46:12.987707 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "d4a4b9786fa0406fa70321ebc813e05e"
format_stamp: "Formatted at 2026-08-06 11:46:12 on dist-test-slave-45dx"
server_key: "1de8e5ceee3345d06a8913e066eef344"
server_key_iv: "e247a4b93d13f86261eb4d1e986d0bbc"
server_key_version: "encryptionkey@0"
I20260806 11:46:12.991350 32702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 11:46:12.993660 32726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:12.994793 32702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 11:46:12.994947 32702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "d4a4b9786fa0406fa70321ebc813e05e"
format_stamp: "Formatted at 2026-08-06 11:46:12 on dist-test-slave-45dx"
server_key: "1de8e5ceee3345d06a8913e066eef344"
server_key_iv: "e247a4b93d13f86261eb4d1e986d0bbc"
server_key_version: "encryptionkey@0"
I20260806 11:46:12.995059 32702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:13.024706 32702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:13.027556 32702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:13.027756 32702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:13.034992 32702 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:33493
I20260806 11:46:13.035056 328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:33493 every 8 connection(s)
I20260806 11:46:13.035983 32702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:13.038823 330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:13.042045 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 32702
I20260806 11:46:13.042213 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:13.042537 452 external_mini_cluster.cc:1428] Setting key 37c2cfe4c4196ffa40a339ca4cc4d96e
I20260806 11:46:13.045058 330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e: Bootstrap starting.
I20260806 11:46:13.047573 330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:13.048506 330 log.cc:824] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:13.050707 330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e: No bootstrap required, opened a new log
I20260806 11:46:13.051568 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:13.043917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42437 (local address 127.0.113.62:33493)
0806 11:46:13.044381 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:13.044397 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:13.044433 (+ 36us) server_negotiation.cc:407] Connection header received
0806 11:46:13.045054 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:13.045070 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:13.045455 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:13.045740 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:13.046127 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:13.047106 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:13.048373 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:13.048926 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:13.049113 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:13.049879 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:13.049884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:13.049897 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:13.050282 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:13.050338 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:13.050361 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:13.050437 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:13.050611 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 11:46:13.050835 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260806 11:46:13.054996 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:13.055480 330 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a4b9786fa0406fa70321ebc813e05e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33493 } }
I20260806 11:46:13.055796 330 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:13.055856 330 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a4b9786fa0406fa70321ebc813e05e, State: Initialized, Role: FOLLOWER
I20260806 11:46:13.056368 330 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a4b9786fa0406fa70321ebc813e05e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33493 } }
I20260806 11:46:13.056567 330 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:13.056644 330 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:13.056748 330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:13.058000 330 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a4b9786fa0406fa70321ebc813e05e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33493 } }
I20260806 11:46:13.058374 330 leader_election.cc:302] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a4b9786fa0406fa70321ebc813e05e; no voters:
I20260806 11:46:13.058734 330 leader_election.cc:288] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:13.058809 344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:13.059019 344 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [term 1 LEADER]: Becoming Leader. State: Replica: d4a4b9786fa0406fa70321ebc813e05e, State: Running, Role: LEADER
I20260806 11:46:13.059311 344 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a4b9786fa0406fa70321ebc813e05e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33493 } }
I20260806 11:46:13.060024 330 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:13.060549 344 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a4b9786fa0406fa70321ebc813e05e. Latest consensus state: current_term: 1 leader_uuid: "d4a4b9786fa0406fa70321ebc813e05e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a4b9786fa0406fa70321ebc813e05e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33493 } } }
I20260806 11:46:13.060739 344 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:13.060540 348 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a4b9786fa0406fa70321ebc813e05e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a4b9786fa0406fa70321ebc813e05e" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 33493 } } }
I20260806 11:46:13.061111 348 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:13.061730 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:13.057365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55545 (local address 127.0.113.62:33493)
0806 11:46:13.057486 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:13.057491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:13.057748 (+ 257us) server_negotiation.cc:407] Connection header received
0806 11:46:13.057883 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:13.057886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:13.057924 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:13.057980 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:13.058291 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:13.058886 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:13.059823 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:13.060071 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:13.060181 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:13.061236 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:13.061241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:13.061245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:13.061261 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:13.061294 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:13.061308 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:13.061376 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:13.061539 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 11:46:13.061591 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 11:46:13.065193 372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:13.065264 372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:13.065387 354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:13.066080 354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:13.071708 354 catalog_manager.cc:1347] Generated new cluster ID: 252acb28b3554c4fb8fc3b6fa2379e10
I20260806 11:46:13.071775 354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:13.092427 354 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:13.093695 354 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:13.100497 354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a4b9786fa0406fa70321ebc813e05e: Generated new TSK 0
I20260806 11:46:13.101267 354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:13.109377 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 32702
2026-08-06T11:46:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1631 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:13 dist-test-slave-45dx krb5kdc[385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:13 dist-test-slave-45dx krb5kdc[385](info): set up 2 sockets
Aug 06 11:46:13 dist-test-slave-45dx krb5kdc[385](info): commencing operation
krb5kdc: starting...
W20260806 11:46:14.354854 452 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:14 dist-test-slave-45dx krb5kdc[385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:46:14.488229 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:34505
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:34505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:14.601024 403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:14.601336 403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:14.601414 403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:14.605436 403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:14.605511 403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:14.605551 403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:14.605609 403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:14.605646 403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:14.610816 403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:34505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:34505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:14.611927 403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:14.613062 403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:14.621958 410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:14.621963 409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:14.622126 403 server_base.cc:1034] running on GCE node
W20260806 11:46:14.621872 414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:14.622941 403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:14.624148 403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:14.625324 403 hybrid_clock.cc:648] HybridClock initialized: now 1786016774625287 us; error 45 us; skew 500 ppm
Aug 06 11:46:14 dist-test-slave-45dx krb5kdc[385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:14.628228 403 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:46:14.629206 403 webserver.cc:460] Webserver started at http://127.0.113.62:40687/ using document root <none> and password file <none>
I20260806 11:46:14.629771 403 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:14.629832 403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:14.630009 403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:14.632138 403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "6209d93e365e40f48bf99429f4bc5fa4"
format_stamp: "Formatted at 2026-08-06 11:46:14 on dist-test-slave-45dx"
server_key: "dfa1ca866ba144a4cedb26d187b9933c"
server_key_iv: "8949a2482bc4bee3d64f3a353f9205c1"
server_key_version: "encryptionkey@0"
I20260806 11:46:14.632771 403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "6209d93e365e40f48bf99429f4bc5fa4"
format_stamp: "Formatted at 2026-08-06 11:46:14 on dist-test-slave-45dx"
server_key: "dfa1ca866ba144a4cedb26d187b9933c"
server_key_iv: "8949a2482bc4bee3d64f3a353f9205c1"
server_key_version: "encryptionkey@0"
I20260806 11:46:14.636632 403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 11:46:14.639034 425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:14.640209 403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:46:14.640372 403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "6209d93e365e40f48bf99429f4bc5fa4"
format_stamp: "Formatted at 2026-08-06 11:46:14 on dist-test-slave-45dx"
server_key: "dfa1ca866ba144a4cedb26d187b9933c"
server_key_iv: "8949a2482bc4bee3d64f3a353f9205c1"
server_key_version: "encryptionkey@0"
I20260806 11:46:14.640478 403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:14.645061 403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:14.647852 403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:14.648044 403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:14.655041 403 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:34505
I20260806 11:46:14.655067 509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:34505 every 8 connection(s)
I20260806 11:46:14.655942 403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:14.658775 510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:14.664239 510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4: Bootstrap starting.
I20260806 11:46:14.665211 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 403
I20260806 11:46:14.665360 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:14.665649 452 external_mini_cluster.cc:1428] Setting key f58be0ac418b6e8ee4f10cfbad93b916
I20260806 11:46:14.666932 510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:14.667689 510 log.cc:824] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:14.669727 510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4: No bootstrap required, opened a new log
I20260806 11:46:14.670642 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:14.666745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39229 (local address 127.0.113.62:34505)
0806 11:46:14.667249 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:14.667265 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:14.667300 (+ 35us) server_negotiation.cc:407] Connection header received
0806 11:46:14.667983 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:14.668008 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:14.668278 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:14.668665 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:14.669477 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:14.669482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:14.669493 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:14.669506 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:14.669561 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:14.669580 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:14.669697 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:14.669797 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 11:46:14.669961 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":150,"threads_started":1}
I20260806 11:46:14.673552 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:14.673945 510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6209d93e365e40f48bf99429f4bc5fa4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34505 } }
I20260806 11:46:14.674189 510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 11:46:14.674247 510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6209d93e365e40f48bf99429f4bc5fa4, State: Initialized, Role: FOLLOWER
I20260806 11:46:14.674638 510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6209d93e365e40f48bf99429f4bc5fa4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34505 } }
I20260806 11:46:14.674885 510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 11:46:14.675002 510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 11:46:14.675124 510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:14.676364 510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6209d93e365e40f48bf99429f4bc5fa4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34505 } }
I20260806 11:46:14.676759 510 leader_election.cc:302] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6209d93e365e40f48bf99429f4bc5fa4; no voters:
I20260806 11:46:14.677107 510 leader_election.cc:288] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 11:46:14.677214 525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:14.677461 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:14.675675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45127 (local address 127.0.113.62:34505)
0806 11:46:14.675809 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:14.675813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:14.676035 (+ 222us) server_negotiation.cc:407] Connection header received
0806 11:46:14.676173 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:14.676175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:14.676203 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:14.676267 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:14.677019 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:14.677022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:14.677024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 11:46:14.677034 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:14.677058 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 11:46:14.677068 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:14.677176 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:14.677293 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 11:46:14.677335 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:46:14.677815 525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [term 1 LEADER]: Becoming Leader. State: Replica: 6209d93e365e40f48bf99429f4bc5fa4, State: Running, Role: LEADER
I20260806 11:46:14.678190 525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6209d93e365e40f48bf99429f4bc5fa4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34505 } }
I20260806 11:46:14.678478 510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:14.679852 529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6209d93e365e40f48bf99429f4bc5fa4. Latest consensus state: current_term: 1 leader_uuid: "6209d93e365e40f48bf99429f4bc5fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6209d93e365e40f48bf99429f4bc5fa4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34505 } } }
I20260806 11:46:14.680001 529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:14.680362 527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6209d93e365e40f48bf99429f4bc5fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6209d93e365e40f48bf99429f4bc5fa4" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 34505 } } }
I20260806 11:46:14.680459 527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4 [sys.catalog]: This master's current role is: LEADER
W20260806 11:46:14.683684 549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:14.683776 549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:14.683846 545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:14.684664 545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:14.689970 545 catalog_manager.cc:1347] Generated new cluster ID: 0888d455730244f89f9e11e8f7121666
I20260806 11:46:14.690039 545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:14.714856 545 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:14.715221 545 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:14.723471 545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6209d93e365e40f48bf99429f4bc5fa4: Generated new TSK 0
I20260806 11:46:14.724191 545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 11:46:14.729202 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 403
2026-08-06T11:46:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1614 ms)
[----------] 18 tests from AuthTokenIssuingTest (29089 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:14 dist-test-slave-45dx krb5kdc[557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:14 dist-test-slave-45dx krb5kdc[557](info): set up 2 sockets
Aug 06 11:46:14 dist-test-slave-45dx krb5kdc[557](info): commencing operation
krb5kdc: starting...
W20260806 11:46:15.964071 452 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:46:16.099624 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36679
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36679,127.0.113.61:39973,127.0.113.60:37893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:16.214665 573 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 11:46:16.214973 573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:16.214998 573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:16.215034 573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:16.219031 573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:16.219097 573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:16.219116 573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:16.219132 573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:16.219147 573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:16.224574 573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36679,127.0.113.61:39973,127.0.113.60:37893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:36679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:16.225666 573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:16.226720 573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:16.235390 580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:16.235390 579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:16.235394 582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:16.235435 573 server_base.cc:1034] running on GCE node
I20260806 11:46:16.236251 573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:16.237413 573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:16.238585 573 hybrid_clock.cc:648] HybridClock initialized: now 1786016776238557 us; error 40 us; skew 500 ppm
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:16.241536 573 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:46:16.242491 573 webserver.cc:460] Webserver started at http://127.0.113.62:46239/ using document root <none> and password file <none>
I20260806 11:46:16.243060 573 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:16.243124 573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:16.243371 573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:16.245520 573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b"
format_stamp: "Formatted at 2026-08-06 11:46:16 on dist-test-slave-45dx"
server_key: "e8611babc7786f452c054d8f5c1aa7c0"
server_key_iv: "bb2b9b6ebbfec42ff16171cc196da7f9"
server_key_version: "encryptionkey@0"
I20260806 11:46:16.246111 573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b"
format_stamp: "Formatted at 2026-08-06 11:46:16 on dist-test-slave-45dx"
server_key: "e8611babc7786f452c054d8f5c1aa7c0"
server_key_iv: "bb2b9b6ebbfec42ff16171cc196da7f9"
server_key_version: "encryptionkey@0"
I20260806 11:46:16.249965 573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 11:46:16.252379 602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:16.253593 573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:46:16.253734 573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b"
format_stamp: "Formatted at 2026-08-06 11:46:16 on dist-test-slave-45dx"
server_key: "e8611babc7786f452c054d8f5c1aa7c0"
server_key_iv: "bb2b9b6ebbfec42ff16171cc196da7f9"
server_key_version: "encryptionkey@0"
I20260806 11:46:16.253865 573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:16.270774 573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:16.273767 573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:16.273984 573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:16.281165 573 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:36679
I20260806 11:46:16.281458 667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:36679 every 8 connection(s)
I20260806 11:46:16.282161 573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:16.285003 668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:16.287082 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 573
I20260806 11:46:16.287215 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:16.287528 452 external_mini_cluster.cc:1428] Setting key c24b3181ed52456f062f67a576308dea
I20260806 11:46:16.289902 668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.61@KRBTEST.COM" created.
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:16.306008 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.293463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.0.113.62:36679)
0806 11:46:16.293752 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.293764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.293821 (+ 57us) server_negotiation.cc:407] Connection header received
0806 11:46:16.294307 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.294312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.294484 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.294663 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.295510 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.296507 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.297509 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.297967 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.298150 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.301043 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.301049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.301059 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.301079 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.302762 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.303307 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.303313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.303315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.303428 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.303802 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.303805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.303806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.304211 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.304383 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.304946 (+ 563us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.305137 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260806 11:46:16.306015 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.292407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36679 (local address 127.0.0.1:46722)
0806 11:46:16.293053 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.293098 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:16.294014 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:16.294662 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:16.294667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:16.295121 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:16.295365 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.295378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.296636 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.296639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.297390 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.297400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.298131 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.298135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.298226 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.298835 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:16.298852 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:16.300872 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:16.302890 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.302900 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.302910 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.303180 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.303529 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.303535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.303540 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.303700 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.304356 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:16.304366 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:16.304596 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 11:46:16.304816 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 11:46:16.305276 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":61,"thread_start_us":155,"threads_started":1}
I20260806 11:46:16.308454 668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:16.309257 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.308928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39973 (local address 127.0.0.1:51240)
0806 11:46:16.309083 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.309179 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 11:46:16.309528 606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
I20260806 11:46:16.310379 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.310191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39973 (local address 127.0.0.1:51254)
0806 11:46:16.310283 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.310327 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 11:46:16.310599 668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.61:39973: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab.
Entry for principal kudu/127.0.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab.
I20260806 11:46:16.317658 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.61:39973
--webserver_interface=127.0.113.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab
--principal=kudu/127.0.113.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36679,127.0.113.61:39973,127.0.113.60:37893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:46:16.359861 668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } attempt: 1
I20260806 11:46:16.360898 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.360460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39973 (local address 127.0.0.1:51266)
0806 11:46:16.360681 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.360778 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 11:46:16.361730 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.361447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39973 (local address 127.0.0.1:51274)
0806 11:46:16.361576 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.361643 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 11:46:16.361891 668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.61:39973: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
I20260806 11:46:16.430068 668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } attempt: 2
I20260806 11:46:16.431027 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.430687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39973 (local address 127.0.0.1:51286)
0806 11:46:16.430855 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.430928 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 11:46:16.431778 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.431561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39973 (local address 127.0.0.1:51288)
0806 11:46:16.431668 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.431718 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 11:46:16.431949 668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.61:39973: Network error: Client connection negotiation failed: client connection to 127.0.113.61:39973: connect: Connection refused (error 111)
W20260806 11:46:16.433706 679 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 11:46:16.433993 679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:16.434039 679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:16.434121 679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:16.438180 679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:16.438232 679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:16.438251 679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:16.438304 679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:16.438335 679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:16.443751 679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36679,127.0.113.61:39973,127.0.113.60:37893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab
--principal=kudu/127.0.113.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.61:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.113.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:16.444900 679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:16.445919 679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:16.454999 685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:16.454999 686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:16.455009 688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:16.455255 679 server_base.cc:1034] running on GCE node
I20260806 11:46:16.455935 679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:16.457188 679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:16.458348 679 hybrid_clock.cc:648] HybridClock initialized: now 1786016776458332 us; error 26 us; skew 500 ppm
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:16.461177 679 init.cc:377] Logged in from keytab as kudu/127.0.113.61@KRBTEST.COM (short username kudu)
I20260806 11:46:16.462118 679 webserver.cc:460] Webserver started at http://127.0.113.61:35889/ using document root <none> and password file <none>
I20260806 11:46:16.462693 679 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:16.462785 679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:16.463022 679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:16.465137 679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/data/instance:
uuid: "99b24c356dd94903b5244ceeb27b95b3"
format_stamp: "Formatted at 2026-08-06 11:46:16 on dist-test-slave-45dx"
server_key: "815c050abbf36f3a3b1e651845a58efe"
server_key_iv: "f05a052e0c7519c00c13edc58aa474e8"
server_key_version: "encryptionkey@0"
I20260806 11:46:16.465690 679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/wal/instance:
uuid: "99b24c356dd94903b5244ceeb27b95b3"
format_stamp: "Formatted at 2026-08-06 11:46:16 on dist-test-slave-45dx"
server_key: "815c050abbf36f3a3b1e651845a58efe"
server_key_iv: "f05a052e0c7519c00c13edc58aa474e8"
server_key_version: "encryptionkey@0"
I20260806 11:46:16.469492 679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 11:46:16.471879 697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:16.473135 679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:46:16.473287 679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/wal
uuid: "99b24c356dd94903b5244ceeb27b95b3"
format_stamp: "Formatted at 2026-08-06 11:46:16 on dist-test-slave-45dx"
server_key: "815c050abbf36f3a3b1e651845a58efe"
server_key_iv: "f05a052e0c7519c00c13edc58aa474e8"
server_key_version: "encryptionkey@0"
I20260806 11:46:16.473405 679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:16.486653 679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:16.489717 679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:16.489949 679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:16.497242 679 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.61:39973
I20260806 11:46:16.497539 772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.61:39973 every 8 connection(s)
I20260806 11:46:16.498270 679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
I20260806 11:46:16.501223 773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:16.505542 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 679
I20260806 11:46:16.505679 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-1/wal/instance
I20260806 11:46:16.505337 773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:16.506001 452 external_mini_cluster.cc:1428] Setting key ab762f2091d9451011344f326f8fa4d4
WARNING: no policy specified for kudu/127.0.113.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.60@KRBTEST.COM" created.
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:16.523150 668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } attempt: 3
I20260806 11:46:16.523707 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.507754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.0.113.62:36679)
0806 11:46:16.507929 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.507934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.509083 (+ 1149us) server_negotiation.cc:407] Connection header received
0806 11:46:16.510052 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.510056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.510122 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.510226 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.511489 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.512029 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.513214 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.513465 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.513579 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.519052 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.519060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.519066 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.519109 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.520893 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.521661 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.521664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.521665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.521757 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.522224 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.522227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.522228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.522444 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.522522 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.523358 (+ 836us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.523518 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 11:46:16.524873 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.508279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36679 (local address 127.0.0.1:46726)
0806 11:46:16.508904 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.508974 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:16.509837 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:16.510398 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:16.510406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:16.510767 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:16.511339 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.511354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.512166 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.512169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.513088 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.513098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.513625 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.513628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.513795 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.514425 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:16.514444 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:16.518820 (+ 4376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:16.521039 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.521048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.521069 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.521507 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.521875 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.521881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.521886 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.522111 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.522555 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:16.522564 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:16.523001 (+ 437us) client_negotiation.cc:769] Sending connection context
0806 11:46:16.523245 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 11:46:16.523679 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":171,"threads_started":1}
I20260806 11:46:16.526752 773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
I20260806 11:46:16.536669 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.523692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39973 (local address 127.0.0.1:51292)
0806 11:46:16.523850 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.523869 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:16.523954 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:16.524973 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:16.524979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:16.525186 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:16.525354 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.525365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.526504 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.526510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.527138 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.527151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.528038 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.528042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.528162 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.528694 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:16.528709 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:16.532288 (+ 3579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:16.534655 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.534658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.534660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.534959 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.535338 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.535342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.535347 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.535463 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.535998 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:16.536002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:16.536129 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 11:46:16.536264 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 11:46:16.536491 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 11:46:16.536823 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.524096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.0.113.61:39973)
0806 11:46:16.524529 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.524542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.524565 (+ 23us) server_negotiation.cc:407] Connection header received
0806 11:46:16.524607 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.524612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.524807 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.524965 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.525495 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.526380 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.527263 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.527909 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.528044 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.532446 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.532457 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.532471 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.532495 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.534505 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.535108 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.535115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.535117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.535230 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.535568 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.535571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.535572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.535876 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.536038 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.536409 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.536581 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":220,"threads_started":1}
I20260806 11:46:16.537428 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.527310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39973 (local address 127.0.0.1:51298)
0806 11:46:16.527507 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.527522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:16.527599 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:16.527886 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:16.527888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:16.528072 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:16.528176 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.528183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.529269 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.529273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.529822 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.529832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.530304 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.530307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.530385 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.530901 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:16.530914 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:16.533313 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:16.535892 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.535895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.535897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.536084 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.536434 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.536437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.536438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.536523 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.537011 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:16.537015 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:16.537099 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 11:46:16.537202 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 11:46:16.537322 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 11:46:16.537914 779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.527361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.0.113.61:39973)
0806 11:46:16.527696 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.527704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.527715 (+ 11us) server_negotiation.cc:407] Connection header received
0806 11:46:16.527749 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.527751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.527781 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.527876 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.528339 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.529173 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.529963 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.530196 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.530287 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.533454 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.533457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.533459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.533512 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.535783 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.536202 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.536207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.536212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.536341 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.536625 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.536628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.536630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.536854 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.536970 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.537700 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.537791 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260806 11:46:16.539059 773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:16.539088 668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab.
Entry for principal kudu/127.0.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab.
I20260806 11:46:16.539839 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.539592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58086)
0806 11:46:16.539706 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.539763 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 11:46:16.539958 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.539607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58082)
0806 11:46:16.539793 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.539875 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260806 11:46:16.540148 700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
I20260806 11:46:16.540774 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.540388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58098)
0806 11:46:16.540643 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.540693 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260806 11:46:16.541055 668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:37893: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
I20260806 11:46:16.541110 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.60:37893
--webserver_interface=127.0.113.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab
--principal=kudu/127.0.113.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:36679,127.0.113.61:39973,127.0.113.60:37893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:46:16.541317 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.540820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58110)
0806 11:46:16.541037 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.541089 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
W20260806 11:46:16.541453 773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:37893: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
I20260806 11:46:16.572254 668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } attempt: 1
I20260806 11:46:16.573251 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.572877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58116)
0806 11:46:16.573073 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.573150 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 11:46:16.574007 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.573769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58122)
0806 11:46:16.573888 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.573941 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 11:46:16.574150 668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:37893: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
I20260806 11:46:16.590754 773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } attempt: 1
I20260806 11:46:16.591663 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.591356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58124)
0806 11:46:16.591512 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.591583 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 11:46:16.592476 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.592226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58134)
0806 11:46:16.592370 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.592422 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 11:46:16.592631 773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:37893: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
I20260806 11:46:16.649338 668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } attempt: 2
I20260806 11:46:16.650282 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.649935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58144)
0806 11:46:16.650112 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.650184 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 11:46:16.651595 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.651356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58150)
0806 11:46:16.651467 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.651521 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 11:46:16.651791 668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:37893: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
I20260806 11:46:16.660794 773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } attempt: 2
W20260806 11:46:16.660794 783 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 11:46:16.661150 783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:16.661209 783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:16.661285 783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 11:46:16.661623 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.661355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58158)
0806 11:46:16.661478 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.661542 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 11:46:16.662313 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.662105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58168)
0806 11:46:16.662206 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.662254 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 11:46:16.662458 773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:37893: Network error: Client connection negotiation failed: client connection to 127.0.113.60:37893: connect: Connection refused (error 111)
W20260806 11:46:16.665573 783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:16.665629 783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:16.665648 783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:16.665664 783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:16.665711 783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:16.671056 783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:36679,127.0.113.61:39973,127.0.113.60:37893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab
--principal=kudu/127.0.113.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.60:37893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.113.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:16.672179 783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:16.673264 783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:16.682760 795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:16.682766 798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:16.682996 783 server_base.cc:1034] running on GCE node
W20260806 11:46:16.682766 790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:16.683678 783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:16.684952 783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:16.686144 783 hybrid_clock.cc:648] HybridClock initialized: now 1786016776686127 us; error 29 us; skew 500 ppm
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:16.689518 783 init.cc:377] Logged in from keytab as kudu/127.0.113.60@KRBTEST.COM (short username kudu)
I20260806 11:46:16.690752 783 webserver.cc:460] Webserver started at http://127.0.113.60:35407/ using document root <none> and password file <none>
I20260806 11:46:16.691568 783 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:16.691668 783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:16.691946 783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:16.694973 783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/data/instance:
uuid: "4b9f3395996148f583c2cedda3ef2c83"
format_stamp: "Formatted at 2026-08-06 11:46:16 on dist-test-slave-45dx"
server_key: "c84e6fea6484f1fa2c8f30b2d8b5a651"
server_key_iv: "3f5b243a832d4e5ac34f0e5401a52281"
server_key_version: "encryptionkey@0"
I20260806 11:46:16.695752 783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/wal/instance:
uuid: "4b9f3395996148f583c2cedda3ef2c83"
format_stamp: "Formatted at 2026-08-06 11:46:16 on dist-test-slave-45dx"
server_key: "c84e6fea6484f1fa2c8f30b2d8b5a651"
server_key_iv: "3f5b243a832d4e5ac34f0e5401a52281"
server_key_version: "encryptionkey@0"
I20260806 11:46:16.701045 783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 11:46:16.704471 807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:16.705842 783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 11:46:16.705998 783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/wal
uuid: "4b9f3395996148f583c2cedda3ef2c83"
format_stamp: "Formatted at 2026-08-06 11:46:16 on dist-test-slave-45dx"
server_key: "c84e6fea6484f1fa2c8f30b2d8b5a651"
server_key_iv: "3f5b243a832d4e5ac34f0e5401a52281"
server_key_version: "encryptionkey@0"
I20260806 11:46:16.706131 783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:16.737583 783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:16.741473 783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:16.741719 783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:16.749118 783 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.60:37893
I20260806 11:46:16.749437 870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.60:37893 every 8 connection(s)
I20260806 11:46:16.750253 783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
I20260806 11:46:16.750999 668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } attempt: 3
I20260806 11:46:16.753587 871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:16.753612 773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } attempt: 3
I20260806 11:46:16.758785 871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:16.759371 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 783
I20260806 11:46:16.759526 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786016488822243-452-0/minicluster-data/master-2/wal/instance
I20260806 11:46:16.759829 452 external_mini_cluster.cc:1428] Setting key e26445c04eaedbd006a51a98f29f8c7b
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
I20260806 11:46:16.766589 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.751560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58184)
0806 11:46:16.751733 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.751752 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:16.751865 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:16.754888 (+ 3023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:16.754891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:16.755104 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:16.755248 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.755257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.756737 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.756741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.757563 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.757576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.758521 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.758525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.758616 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.760256 (+ 1640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:16.760272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:16.762054 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:16.764338 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.764341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.764344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.764629 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.765029 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.765033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.765034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.765134 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.766015 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:16.766018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:16.766138 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 11:46:16.766222 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 11:46:16.766384 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 11:46:16.766979 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.754138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58198)
0806 11:46:16.754249 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.754264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:16.754398 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:16.754836 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:16.754839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:16.755080 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:16.755237 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.755244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.756689 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.756694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.757595 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.757607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.758491 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.758495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.758593 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.759317 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:16.759334 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:16.761006 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:16.764404 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.764407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.764410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.764672 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.765046 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.765049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.765051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.765156 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.766008 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:16.766011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:16.766114 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 11:46:16.766574 (+ 460us) client_negotiation.cc:240] Negotiation successful
0806 11:46:16.766785 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
I20260806 11:46:16.768524 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.754179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.0.113.60:37893)
0806 11:46:16.754526 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.754535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.754553 (+ 18us) server_negotiation.cc:407] Connection header received
0806 11:46:16.754585 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.754586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.754618 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.754920 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.755374 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.756560 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.757734 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.758350 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.758537 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.761216 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.761223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.761240 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.761274 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.764221 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.764805 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.764810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.764812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.764934 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.765271 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.765274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.765276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.765881 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.767616 (+ 1735us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.767694 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.767804 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":278,"spinlock_wait_cycles":37504,"thread_start_us":133,"threads_started":1}
I20260806 11:46:16.768522 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.752342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.0.113.60:37893)
0806 11:46:16.752800 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.752820 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.752866 (+ 46us) server_negotiation.cc:407] Connection header received
0806 11:46:16.753797 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.753822 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.754536 (+ 714us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.755372 (+ 836us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.755533 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.756561 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.757689 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.758351 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.758881 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.762199 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.762203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.762205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.762230 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.764181 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.764767 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.764773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.764779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.764918 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.765253 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.765256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.765258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.765881 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.766841 (+ 960us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.767033 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.767357 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":300,"spinlock_wait_cycles":3840,"thread_start_us":122,"threads_started":1}
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:16.773693 773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3: Bootstrap starting.
I20260806 11:46:16.773940 668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b: Bootstrap starting.
I20260806 11:46:16.774077 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.760927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.0.113.60:37893)
0806 11:46:16.761830 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.761839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.761856 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:46:16.761893 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.761896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.761951 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.762046 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.763171 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.764009 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.765253 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.765576 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.765677 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.768717 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.768720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.768723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.768746 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.771531 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.772065 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.772069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.772070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.772164 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.772540 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.772544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.772547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.772816 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.773008 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.773737 (+ 729us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.773854 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"spinlock_wait_cycles":9600,"thread_start_us":131,"threads_started":1}
I20260806 11:46:16.775377 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.761280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46728 (local address 127.0.113.62:36679)
0806 11:46:16.761407 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.761412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.762275 (+ 863us) server_negotiation.cc:407] Connection header received
0806 11:46:16.762502 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.762506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.762557 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.762614 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.764828 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.765569 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.768146 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.768478 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.768572 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.771591 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.771596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.771598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.771626 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.773346 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.773898 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.773902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.773903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.773984 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.774303 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.774306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.774308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.774589 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.774665 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.775115 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.775239 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 11:46:16.776245 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.760937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:36679 (local address 127.0.0.1:46728)
0806 11:46:16.762136 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.762196 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:16.762379 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:16.762811 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:16.762819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:16.764485 (+ 1666us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:16.764689 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.764700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.765708 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.765711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.768006 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.768016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.768963 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.768967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.769073 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.769617 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:16.769631 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:16.771411 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:16.773476 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.773485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.773490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.773782 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.774101 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.774103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.774106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.774196 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.774749 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:16.774755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:16.774870 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 11:46:16.775817 (+ 947us) client_negotiation.cc:240] Negotiation successful
0806 11:46:16.776024 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":115,"threads_started":1}
I20260806 11:46:16.776877 773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:16.777359 668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:16.777319 871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:16.777890 773 log.cc:824] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:16.778409 668 log.cc:824] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:16.780684 773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3: No bootstrap required, opened a new log
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:16.782215 668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b: No bootstrap required, opened a new log
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
I20260806 11:46:16.785614 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.775221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.0.113.62:36679)
0806 11:46:16.775551 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.775555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.775567 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:46:16.775600 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.775603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.775639 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.775739 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.776535 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.777253 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.778363 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.778684 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.778765 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.781651 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.781655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.781658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.781682 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.783425 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.783920 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.783936 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.783938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.784026 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.784408 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.784411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.784413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.784646 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.784821 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.785257 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.785402 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20260806 11:46:16.785673 773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } }
I20260806 11:46:16.786509 773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b24c356dd94903b5244ceeb27b95b3, State: Initialized, Role: FOLLOWER
I20260806 11:46:16.786878 773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } }
I20260806 11:46:16.786906 668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } }
I20260806 11:46:16.787326 668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687ba1ac6adc48a3bb1e590ae08b4f7b, State: Initialized, Role: FOLLOWER
I20260806 11:46:16.787781 668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } }
I20260806 11:46:16.788123 779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.778109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.0.113.61:39973)
0806 11:46:16.778239 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.778244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.778260 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:46:16.778303 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.778305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.778349 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.778441 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.778910 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.779690 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.780900 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.781300 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.781387 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.784281 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.784287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.784294 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.784346 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.786155 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.786666 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.786670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.786671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.786759 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.787098 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.787101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.787103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.787368 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.787458 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.787834 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.787962 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 11:46:16.788643 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.777771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:39973 (local address 127.0.0.1:51308)
0806 11:46:16.778135 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.778148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:16.778218 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:16.778454 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:16.778457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:16.778619 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:16.778754 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.778761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.779820 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.779827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.780743 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.780759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.781569 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.781572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.781672 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.782368 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:16.782383 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:16.784129 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:16.786283 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.786287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.786288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.786543 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.786877 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.786880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.786882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.786984 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.787543 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:16.787546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:16.787644 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 11:46:16.788267 (+ 623us) client_negotiation.cc:240] Negotiation successful
0806 11:46:16.788458 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":107,"threads_started":1}
I20260806 11:46:16.789373 871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:16.789688 773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:16.789688 668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:16.793001 890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.793243 890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:16.793529 887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.793736 887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [sys.catalog]: This master's current role is: FOLLOWER
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
W20260806 11:46:16.796859 925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:16.796947 925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:16.798992 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.790037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.0.113.60:37893)
0806 11:46:16.790134 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.790139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.790191 (+ 52us) server_negotiation.cc:407] Connection header received
0806 11:46:16.790320 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.790322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.790353 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.790392 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.790870 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.791570 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.792651 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.793008 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.793085 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.795462 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.795466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.795469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.795493 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.797253 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.797730 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.797733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.797735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.797816 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.798104 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.798109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.798110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.798325 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.798397 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.798711 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.798837 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 11:46:16.799456 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.789821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:37893 (local address 127.0.0.1:58216)
0806 11:46:16.790132 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:16.790146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:16.790241 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:16.790501 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:16.790503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:16.790673 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:16.790772 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.790779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.791688 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.791692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.792529 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:16.792540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.793210 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.793214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:16.793283 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.793743 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:16.793757 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:16.795318 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:16.797367 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.797370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.797375 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.797631 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.797921 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:16.797924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:16.797925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:16.798011 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:16.798466 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:16.798469 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:16.798546 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 11:46:16.799171 (+ 625us) client_negotiation.cc:240] Negotiation successful
0806 11:46:16.799302 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
W20260806 11:46:16.799976 923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:16.800066 923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:16.802400 871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83: Bootstrap starting.
I20260806 11:46:16.800617 679 master_runner.cc:186] Error getting master registration for 127.0.113.60:37893: OK, instance_id { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" instance_seqno: 1786016776747120 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 11:46:16.804557 679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 11:46:16.805374 871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:16.806284 871 log.cc:824] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:16.804801 573 master_runner.cc:186] Error getting master registration for 127.0.113.60:37893: OK, instance_id { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" instance_seqno: 1786016776747120 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 11:46:16.808455 573 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 11:46:16.808744 871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83: No bootstrap required, opened a new log
Aug 06 11:46:16 dist-test-slave-45dx krb5kdc[557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
I20260806 11:46:16.814791 871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } }
I20260806 11:46:16.815080 871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9f3395996148f583c2cedda3ef2c83, State: Initialized, Role: FOLLOWER
I20260806 11:46:16.815156 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.787369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.0.113.61:39973)
0806 11:46:16.787485 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.787492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.787679 (+ 187us) server_negotiation.cc:407] Connection header received
0806 11:46:16.787861 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.787864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.787903 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.787971 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.788850 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.789556 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.790758 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.791121 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.791192 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.810831 (+ 19639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.810836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.810839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.810875 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.812866 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.813508 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.813514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.813517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.813635 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.814008 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.814011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.814013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.814255 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.814437 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.814831 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.814980 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:46:16.815455 871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } }
I20260806 11:46:16.817343 871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:16.818857 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:16.820480 634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46736
I20260806 11:46:16.820812 634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 11:46:16.820957 634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:16.822954 634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } }
W20260806 11:46:16.823565 943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:16.823657 943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:16.823913 930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.824055 930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:16.824555 743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99b24c356dd94903b5244ceeb27b95b3"
I20260806 11:46:16.825560 743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:16.826238 634 leader_election.cc:288] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [CANDIDATE]: Term 1 election: Requested vote from peers 99b24c356dd94903b5244ceeb27b95b3 (127.0.113.61:39973), 4b9f3395996148f583c2cedda3ef2c83 (127.0.113.60:37893)
I20260806 11:46:16.827116 743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687ba1ac6adc48a3bb1e590ae08b4f7b in term 1.
I20260806 11:46:16.827536 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4b9f3395996148f583c2cedda3ef2c83"
I20260806 11:46:16.827768 841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:16.829245 841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687ba1ac6adc48a3bb1e590ae08b4f7b in term 1.
I20260806 11:46:16.829939 604 leader_election.cc:302] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9f3395996148f583c2cedda3ef2c83, 687ba1ac6adc48a3bb1e590ae08b4f7b; no voters:
I20260806 11:46:16.830636 890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:16.830878 890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [term 1 LEADER]: Becoming Leader. State: Replica: 687ba1ac6adc48a3bb1e590ae08b4f7b, State: Running, Role: LEADER
I20260806 11:46:16.831247 890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } }
I20260806 11:46:16.834275 890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687ba1ac6adc48a3bb1e590ae08b4f7b. Latest consensus state: current_term: 1 leader_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.834401 890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:16.836254 783 master_runner.cc:263] No leader master found from master 4b9f3395996148f583c2cedda3ef2c83
I20260806 11:46:16.836359 783 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 11:46:16.840673 952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:16.841748 952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:16.862603 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.835890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.0.113.62:36679)
0806 11:46:16.836013 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.836018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.844385 (+ 8367us) server_negotiation.cc:407] Connection header received
0806 11:46:16.844614 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.844620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.844672 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.844724 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.845788 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.846578 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.848422 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.848774 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.848846 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.856155 (+ 7309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.856159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.856162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.856194 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.858195 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.859936 (+ 1741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.859940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.859942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.860038 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.860368 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.860372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.860374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.860616 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.860732 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.862262 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.862406 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 11:46:16.871665 841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [term 1 FOLLOWER]: Refusing update from remote peer 687ba1ac6adc48a3bb1e590ae08b4f7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:46:16.872411 950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:46:16.876668 930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687ba1ac6adc48a3bb1e590ae08b4f7b. Latest consensus state: current_term: 1 leader_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.876804 930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:16.878561 743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [term 1 FOLLOWER]: Refusing update from remote peer 687ba1ac6adc48a3bb1e590ae08b4f7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:46:16.879187 950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:46:16.883514 887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687ba1ac6adc48a3bb1e590ae08b4f7b. Latest consensus state: current_term: 1 leader_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.883665 887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:16.884678 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.836795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.0.113.61:39973)
0806 11:46:16.836912 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.836916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.847994 (+ 11078us) server_negotiation.cc:407] Connection header received
0806 11:46:16.848170 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.848173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.848224 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.848273 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.851628 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.852453 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.858840 (+ 6387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.859199 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.859281 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.865582 (+ 6301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.865586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.865589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.865624 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.867380 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.869915 (+ 2535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.869920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.869922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.870022 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.877080 (+ 7058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.877084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.877086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.877413 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.877565 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.884325 (+ 6760us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.884499 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 11:46:16.899421 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:16.836294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.0.113.60:37893)
0806 11:46:16.836439 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:16.836444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:16.864290 (+ 27846us) server_negotiation.cc:407] Connection header received
0806 11:46:16.864623 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:16.864626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:16.864680 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:16.864732 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:16.867917 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.868771 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.886359 (+ 17588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:16.886752 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:16.886837 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:16.888780 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:16.888784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:16.888787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:16.888817 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:16.890561 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.892739 (+ 2178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.892743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.892745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.892839 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:16.895101 (+ 2262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:16.895106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:16.895108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:16.895366 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:16.895501 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:16.899083 (+ 3582us) server_negotiation.cc:299] Negotiation successful
0806 11:46:16.899239 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 11:46:16.902100 950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.902253 950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:16.903503 887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.903646 887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:16.904059 957 mvcc.cc:205] Tried to move back new op lower bound from 7315524717999968256 to 7315524717904551936. Current Snapshot: MvccSnapshot[applied={T|T < 7315524717999968256}]
I20260806 11:46:16.905100 890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.905234 890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:16.906478 930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687ba1ac6adc48a3bb1e590ae08b4f7b" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 36679 } } peers { permanent_uuid: "99b24c356dd94903b5244ceeb27b95b3" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 39973 } } peers { permanent_uuid: "4b9f3395996148f583c2cedda3ef2c83" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 37893 } } }
I20260806 11:46:16.906608 930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:16.907095 962 mvcc.cc:205] Tried to move back new op lower bound from 7315524717999968256 to 7315524717904551936. Current Snapshot: MvccSnapshot[applied={T|T < 7315524717999968256}]
I20260806 11:46:16.907289 958 mvcc.cc:205] Tried to move back new op lower bound from 7315524717999968256 to 7315524717904551936. Current Snapshot: MvccSnapshot[applied={T|T < 7315524717999968256}]
I20260806 11:46:16.909648 952 catalog_manager.cc:1347] Generated new cluster ID: 978332a0d4914d11bbbfc7ed66335473
I20260806 11:46:16.909718 952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:16.941538 952 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:16.942832 952 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:16.966473 952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 687ba1ac6adc48a3bb1e590ae08b4f7b: Generated new TSK 0
I20260806 11:46:16.967350 952 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 11:46:17.007002 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.003354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.0.113.61:39973)
0806 11:46:17.003506 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.003511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.005730 (+ 2219us) server_negotiation.cc:407] Connection header received
0806 11:46:17.005978 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.005982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.006039 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.006099 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.006836 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51336: BlockingRecv error: recv got EOF from 127.0.0.1:51336 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 11:46:17.008883 675 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.004410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.0.113.62:36679)
0806 11:46:17.004539 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.004544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.007728 (+ 3184us) server_negotiation.cc:407] Connection header received
0806 11:46:17.007942 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.007946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.008005 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.008062 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.008747 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46752: BlockingRecv error: recv got EOF from 127.0.0.1:46752 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 11:46:17.010829 880 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.005098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.0.113.60:37893)
0806 11:46:17.005215 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.005220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.009729 (+ 4509us) server_negotiation.cc:407] Connection header received
0806 11:46:17.009921 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.009936 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.009993 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.010051 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.010705 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58236: BlockingRecv error: recv got EOF from 127.0.0.1:58236 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 11:46:17.016040 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.014308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.113.61:39973)
0806 11:46:17.014424 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.014428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.014980 (+ 552us) server_negotiation.cc:407] Connection header received
0806 11:46:17.015224 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.015227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.015313 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.015371 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.015916 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51352: BlockingRecv error: recv got EOF from 127.0.0.1:51352 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 11:46:17.021162 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.019770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.0.113.61:39973)
0806 11:46:17.019895 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.019902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.020195 (+ 293us) server_negotiation.cc:407] Connection header received
0806 11:46:17.020462 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.020466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.020515 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.020563 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.021063 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51354: BlockingRecv error: recv got EOF from 127.0.0.1:51354 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 11:46:17.027668 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.025658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.0.113.61:39973)
0806 11:46:17.026457 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.026462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.026729 (+ 267us) server_negotiation.cc:407] Connection header received
0806 11:46:17.026862 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.026865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.026913 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.026989 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.027531 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51360: BlockingRecv error: recv got EOF from 127.0.0.1:51360 (error 108)
Metrics: {"server-negotiator.queue_time_us":718}
W20260806 11:46:17.034807 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.033675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.0.113.61:39973)
0806 11:46:17.033845 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.033848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.033946 (+ 98us) server_negotiation.cc:407] Connection header received
0806 11:46:17.034084 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.034086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.034129 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.034196 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.034697 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51374: BlockingRecv error: recv got EOF from 127.0.0.1:51374 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260806 11:46:17.045579 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.044567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.0.113.61:39973)
0806 11:46:17.044694 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.044697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.044881 (+ 184us) server_negotiation.cc:407] Connection header received
0806 11:46:17.044998 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.045000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.045046 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.045127 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.045488 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51384: BlockingRecv error: recv got EOF from 127.0.0.1:51384 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 11:46:17.064965 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.063775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.0.113.61:39973)
0806 11:46:17.063931 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.063935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.064089 (+ 154us) server_negotiation.cc:407] Connection header received
0806 11:46:17.064180 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.064182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.064234 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.064405 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.064801 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51400: BlockingRecv error: recv got EOF from 127.0.0.1:51400 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 11:46:17.100708 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.099397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.0.113.61:39973)
0806 11:46:17.099561 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.099567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.099731 (+ 164us) server_negotiation.cc:407] Connection header received
0806 11:46:17.099912 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.099915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.099968 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.100054 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.100566 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51416: BlockingRecv error: recv got EOF from 127.0.0.1:51416 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 11:46:17.167966 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.166846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.0.113.61:39973)
0806 11:46:17.166982 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.166986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.167149 (+ 163us) server_negotiation.cc:407] Connection header received
0806 11:46:17.167275 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.167278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.167326 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.167436 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.167860 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51432: BlockingRecv error: recv got EOF from 127.0.0.1:51432 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 11:46:17.299515 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.298186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.0.113.61:39973)
0806 11:46:17.298340 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.298346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.298597 (+ 251us) server_negotiation.cc:407] Connection header received
0806 11:46:17.298725 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.298728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.298778 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.298882 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.299391 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51444: BlockingRecv error: recv got EOF from 127.0.0.1:51444 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 11:46:17.558863 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:17.557847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.0.113.61:39973)
0806 11:46:17.558021 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:17.558025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:17.558043 (+ 18us) server_negotiation.cc:407] Connection header received
0806 11:46:17.558181 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:17.558185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:17.558234 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:17.558318 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:17.558756 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51458: BlockingRecv error: recv got EOF from 127.0.0.1:51458 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 11:46:17.798410 925 catalog_manager.cc:1259] Loaded cluster ID: 978332a0d4914d11bbbfc7ed66335473
I20260806 11:46:17.798547 925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3: loading cluster ID for follower catalog manager: success
I20260806 11:46:17.800926 925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3: acquiring CA information for follower catalog manager: success
I20260806 11:46:17.802203 925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99b24c356dd94903b5244ceeb27b95b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 11:46:17.825101 943 catalog_manager.cc:1259] Loaded cluster ID: 978332a0d4914d11bbbfc7ed66335473
I20260806 11:46:17.825194 943 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83: loading cluster ID for follower catalog manager: success
I20260806 11:46:17.827009 943 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83: acquiring CA information for follower catalog manager: success
I20260806 11:46:17.828293 943 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b9f3395996148f583c2cedda3ef2c83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 11:46:18.077164 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:18.073263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.0.113.61:39973)
0806 11:46:18.073556 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:18.073560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:18.073580 (+ 20us) server_negotiation.cc:407] Connection header received
0806 11:46:18.073735 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:18.073738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:18.073867 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:18.073988 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:18.074238 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.074749 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.075561 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.075782 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.075855 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:18.076154 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:18.076655 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:18.076788 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:18.076896 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 11:46:18.076961 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":112,"threads_started":1}
I20260806 11:46:18.084167 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:18.080463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.0.113.60:37893)
0806 11:46:18.080707 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:18.080712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:18.080727 (+ 15us) server_negotiation.cc:407] Connection header received
0806 11:46:18.080904 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:18.080907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:18.081036 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:18.081159 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:18.081372 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.081902 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.082727 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.082972 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.083082 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:18.083376 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:18.083773 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:18.083904 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:18.083952 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 11:46:18.084007 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260806 11:46:18.091593 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:18.087585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58254 (local address 127.0.113.60:37893)
0806 11:46:18.087691 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:18.087694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:18.087797 (+ 103us) server_negotiation.cc:407] Connection header received
0806 11:46:18.087927 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:18.087930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:18.087975 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:18.088052 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:18.088298 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.089079 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.090199 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.090548 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.090672 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:18.091072 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:18.091162 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:18.091235 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:18.091355 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 11:46:18.091405 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 11:46:18.091616 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:18.087026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.0.113.61:39973)
0806 11:46:18.087162 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:18.087169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:18.087608 (+ 439us) server_negotiation.cc:407] Connection header received
0806 11:46:18.087778 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:18.087781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:18.087832 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:18.087909 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:18.088471 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.088955 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.090282 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.090590 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.090709 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:18.091110 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:18.091182 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:18.091263 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:18.091422 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 11:46:18.091482 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 11:46:18.092526 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:18.087006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.0.113.62:36679)
0806 11:46:18.087341 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:18.087348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:18.087559 (+ 211us) server_negotiation.cc:407] Connection header received
0806 11:46:18.087661 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:18.087664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:18.087820 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:18.087927 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:18.088467 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.089242 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.090325 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:18.090718 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:18.090843 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:18.091258 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:18.091803 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:18.091890 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:18.092290 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 11:46:18.092376 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260806 11:46:18.095606 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 573
I20260806 11:46:18.117055 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 679
I20260806 11:46:18.130535 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 783
2026-08-06T11:46:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3404 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 11:46:18 dist-test-slave-45dx krb5kdc[1084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 11:46:18 dist-test-slave-45dx krb5kdc[1084](info): set up 2 sockets
Aug 06 11:46:18 dist-test-slave-45dx krb5kdc[1084](info): commencing operation
krb5kdc: starting...
W20260806 11:46:19.380955 452 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/test-user.keytab.
Aug 06 11:46:19 dist-test-slave-45dx krb5kdc[1084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T11:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T11:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
Entry for principal kudu/127.0.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab.
I20260806 11:46:19.508010 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:43627
--webserver_interface=127.0.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:43627,127.0.113.61:40225,127.0.113.60:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 11:46:19.621677 1104 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 11:46:19.622105 1104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:19.622167 1104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:19.622248 1104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:19.626318 1104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:19.626374 1104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:19.626392 1104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:19.626410 1104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:19.626428 1104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:19.631536 1104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:43627,127.0.113.61:40225,127.0.113.60:45599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.62.keytab
--principal=kudu/127.0.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.62:43627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:19.632591 1104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:19.633607 1104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:19.642346 1115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:19.642360 1113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:19.642422 1104 server_base.cc:1034] running on GCE node
W20260806 11:46:19.642346 1112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:19.643234 1104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:19.644384 1104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:19.645567 1104 hybrid_clock.cc:648] HybridClock initialized: now 1786016779645554 us; error 25 us; skew 500 ppm
Aug 06 11:46:19 dist-test-slave-45dx krb5kdc[1084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:19.648360 1104 init.cc:377] Logged in from keytab as kudu/127.0.113.62@KRBTEST.COM (short username kudu)
I20260806 11:46:19.649315 1104 webserver.cc:460] Webserver started at http://127.0.113.62:35889/ using document root <none> and password file <none>
I20260806 11:46:19.649827 1104 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:19.649884 1104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:19.650058 1104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:19.652056 1104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/data/instance:
uuid: "5e5a2c25f5bb4c659f329b4886998124"
format_stamp: "Formatted at 2026-08-06 11:46:19 on dist-test-slave-45dx"
server_key: "c7e6667614baf6ecdb4e74e43a8decda"
server_key_iv: "c8dc18dfbf4b7c8ace19206e02c57b4e"
server_key_version: "encryptionkey@0"
I20260806 11:46:19.652627 1104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/wal/instance:
uuid: "5e5a2c25f5bb4c659f329b4886998124"
format_stamp: "Formatted at 2026-08-06 11:46:19 on dist-test-slave-45dx"
server_key: "c7e6667614baf6ecdb4e74e43a8decda"
server_key_iv: "c8dc18dfbf4b7c8ace19206e02c57b4e"
server_key_version: "encryptionkey@0"
I20260806 11:46:19.656243 1104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 11:46:19.658548 1124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:19.659734 1104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 11:46:19.659886 1104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/wal
uuid: "5e5a2c25f5bb4c659f329b4886998124"
format_stamp: "Formatted at 2026-08-06 11:46:19 on dist-test-slave-45dx"
server_key: "c7e6667614baf6ecdb4e74e43a8decda"
server_key_iv: "c8dc18dfbf4b7c8ace19206e02c57b4e"
server_key_version: "encryptionkey@0"
I20260806 11:46:19.659992 1104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:19.687909 1104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:19.691385 1104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:19.691553 1104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:19.698782 1104 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.62:43627
I20260806 11:46:19.699070 1205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.62:43627 every 8 connection(s)
I20260806 11:46:19.699777 1104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/data/info.pb
I20260806 11:46:19.702567 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:19.705030 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 1104
I20260806 11:46:19.705156 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-0/wal/instance
I20260806 11:46:19.705453 452 external_mini_cluster.cc:1428] Setting key edcc4c5c3e90dcc6f1645ece10a7c6f0
I20260806 11:46:19.706663 1206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.61@KRBTEST.COM" created.
Aug 06 11:46:19 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:19.725118 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.709227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43627 (local address 127.0.0.1:38144)
0806 11:46:19.709742 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.709796 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:19.710694 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:19.711605 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:19.711612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:19.711977 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:19.712225 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:19.712238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.714063 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.714065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:19.714766 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:19.714774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.715672 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.715675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:19.715766 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:19.716452 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:19.716468 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:19.719932 (+ 3464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:19.721982 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:19.721992 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:19.722003 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:19.722236 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:19.722564 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:19.722567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:19.722568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:19.722685 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:19.723288 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:19.723295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:19.723453 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 11:46:19.723630 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 11:46:19.724051 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":122,"thread_start_us":147,"threads_started":1}
I20260806 11:46:19.725118 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.709227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.0.113.62:43627)
0806 11:46:19.709730 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:19.709751 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:19.709930 (+ 179us) server_negotiation.cc:407] Connection header received
0806 11:46:19.711029 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:19.711039 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:19.711337 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:19.711562 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:19.712421 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.713936 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.714888 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.715519 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.715716 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:19.720093 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:19.720098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:19.720110 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:19.720131 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:19.721836 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:19.722348 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:19.722353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:19.722355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:19.722467 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:19.722776 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:19.722781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:19.722782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:19.723175 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:19.723319 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:19.723706 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 11:46:19.724004 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":361,"spinlock_wait_cycles":45440,"thread_start_us":210,"threads_started":1}
I20260806 11:46:19.727811 1206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:19.728596 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.728337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:40225 (local address 127.0.0.1:55746)
0806 11:46:19.728459 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.728530 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 11:46:19.728826 1127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
I20260806 11:46:19.729691 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.729489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:40225 (local address 127.0.0.1:55762)
0806 11:46:19.729592 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.729635 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 11:46:19.729843 1206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.61:40225: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab.
Entry for principal kudu/127.0.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab.
I20260806 11:46:19.735343 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.61:40225
--webserver_interface=127.0.113.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab
--principal=kudu/127.0.113.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:43627,127.0.113.61:40225,127.0.113.60:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:46:19.779110 1206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } attempt: 1
I20260806 11:46:19.780085 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.779770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:40225 (local address 127.0.0.1:55774)
0806 11:46:19.779936 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.780003 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 11:46:19.780844 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.780610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:40225 (local address 127.0.0.1:55776)
0806 11:46:19.780717 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.780775 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 11:46:19.781016 1206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.61:40225: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
I20260806 11:46:19.849192 1206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } attempt: 2
I20260806 11:46:19.850111 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.849810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:40225 (local address 127.0.0.1:55788)
0806 11:46:19.849967 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.850032 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 11:46:19.850833 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.850646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:40225 (local address 127.0.0.1:55790)
0806 11:46:19.850738 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.850782 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 11:46:19.851027 1206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.61:40225: Network error: Client connection negotiation failed: client connection to 127.0.113.61:40225: connect: Connection refused (error 111)
W20260806 11:46:19.850910 1217 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 11:46:19.851339 1217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:19.851382 1217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:19.851471 1217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:19.855525 1217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:19.855587 1217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:19.855607 1217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:19.855656 1217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:19.855688 1217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:19.861058 1217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:43627,127.0.113.61:40225,127.0.113.60:45599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.61.keytab
--principal=kudu/127.0.113.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.61:40225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.113.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:19.862166 1217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:19.863260 1217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:19.872153 1226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:19.872159 1224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:19.872318 1217 server_base.cc:1034] running on GCE node
W20260806 11:46:19.872514 1230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:19.873075 1217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:19.874202 1217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:19.875392 1217 hybrid_clock.cc:648] HybridClock initialized: now 1786016779875376 us; error 28 us; skew 500 ppm
Aug 06 11:46:19 dist-test-slave-45dx krb5kdc[1084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:19.878242 1217 init.cc:377] Logged in from keytab as kudu/127.0.113.61@KRBTEST.COM (short username kudu)
I20260806 11:46:19.879153 1217 webserver.cc:460] Webserver started at http://127.0.113.61:46065/ using document root <none> and password file <none>
I20260806 11:46:19.879726 1217 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:19.879815 1217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:19.880049 1217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:19.882238 1217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/data/instance:
uuid: "3df44e94e0a74bde9fc4f73549d9c0fc"
format_stamp: "Formatted at 2026-08-06 11:46:19 on dist-test-slave-45dx"
server_key: "c1f8fbd2079d5e302f858ecc21e70582"
server_key_iv: "0cf231c23bd48872a00af5e66acd6c74"
server_key_version: "encryptionkey@0"
I20260806 11:46:19.882825 1217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/wal/instance:
uuid: "3df44e94e0a74bde9fc4f73549d9c0fc"
format_stamp: "Formatted at 2026-08-06 11:46:19 on dist-test-slave-45dx"
server_key: "c1f8fbd2079d5e302f858ecc21e70582"
server_key_iv: "0cf231c23bd48872a00af5e66acd6c74"
server_key_version: "encryptionkey@0"
I20260806 11:46:19.886530 1217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 11:46:19.888960 1240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:19.890161 1217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 11:46:19.890301 1217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/wal
uuid: "3df44e94e0a74bde9fc4f73549d9c0fc"
format_stamp: "Formatted at 2026-08-06 11:46:19 on dist-test-slave-45dx"
server_key: "c1f8fbd2079d5e302f858ecc21e70582"
server_key_iv: "0cf231c23bd48872a00af5e66acd6c74"
server_key_version: "encryptionkey@0"
I20260806 11:46:19.890416 1217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:19.906662 1217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:19.909825 1217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:19.910074 1217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:19.917097 1217 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.61:40225
I20260806 11:46:19.917388 1311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.61:40225 every 8 connection(s)
I20260806 11:46:19.918093 1217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/data/info.pb
I20260806 11:46:19.920990 1312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:19.922777 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 1217
I20260806 11:46:19.922891 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-1/wal/instance
I20260806 11:46:19.923206 452 external_mini_cluster.cc:1428] Setting key ebd2d1f82db7741a05afa4e60bcd2fa8
I20260806 11:46:19.925203 1312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.113.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.113.60@KRBTEST.COM" created.
Aug 06 11:46:19 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:19.941538 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.927705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.0.113.62:43627)
0806 11:46:19.927893 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:19.927900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:19.928811 (+ 911us) server_negotiation.cc:407] Connection header received
0806 11:46:19.929810 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:19.929817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:19.929896 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:19.930023 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:19.931205 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.932087 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.933349 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.933590 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.933680 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:19.937169 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:19.937173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:19.937175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:19.937203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:19.939102 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:19.939667 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:19.939671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:19.939673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:19.939771 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:19.940103 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:19.940109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:19.940113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:19.940431 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:19.940545 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:19.941098 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 11:46:19.941256 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 11:46:19.942221 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.928084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43627 (local address 127.0.0.1:38154)
0806 11:46:19.928636 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.928692 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:19.929585 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:19.930172 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:19.930179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:19.930537 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:19.931036 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:19.931051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.932224 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.932227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:19.933202 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:19.933213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.933729 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.933731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:19.933873 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:19.934453 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:19.934470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:19.936981 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:19.939233 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:19.939238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:19.939252 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:19.939543 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:19.939854 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:19.939856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:19.939857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:19.939997 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:19.940536 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:19.940542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:19.940802 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 11:46:19.940971 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 11:46:19.941240 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":183,"threads_started":1}
I20260806 11:46:19.942185 1206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } attempt: 3
I20260806 11:46:19.943799 1312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 11:46:19 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
Aug 06 11:46:19 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab.
Entry for principal kudu/127.0.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab.
I20260806 11:46:19.956255 452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
/tmp/dist-test-taskg6qW6_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.60:45599
--webserver_interface=127.0.113.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab
--principal=kudu/127.0.113.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.113.20:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.113.62:43627,127.0.113.61:40225,127.0.113.60:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 11:46:19.956740 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.942604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:40225 (local address 127.0.0.1:55800)
0806 11:46:19.942900 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.942916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:19.943009 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:19.943781 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:19.943784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:19.943986 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:19.944090 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:19.944096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.945611 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.945615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:19.946292 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:19.946301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.946895 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.946898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:19.947018 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:19.947505 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:19.947520 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:19.949959 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:19.953890 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:19.953896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:19.953900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:19.954224 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:19.954656 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:19.954661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:19.954665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:19.954778 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:19.956061 (+ 1283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:19.956068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:19.956189 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 11:46:19.956284 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 11:46:19.956488 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260806 11:46:19.957211 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.944381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:40225 (local address 127.0.0.1:55810)
0806 11:46:19.944560 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.944575 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:19.944648 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:19.944919 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:19.944922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:19.945087 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:19.945160 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:19.945165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.946064 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.946069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:19.946994 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:19.947009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.947551 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.947554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:19.947796 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:19.948380 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:19.948397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:19.950768 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:19.954722 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:19.954724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:19.954727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:19.954978 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:19.955797 (+ 819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:19.955799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:19.955801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:19.955906 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:19.956525 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:19.956531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:19.956656 (+ 125us) client_negotiation.cc:769] Sending connection context
0806 11:46:19.956785 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 11:46:19.956976 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 11:46:19.957690 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.942925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.0.113.61:40225)
0806 11:46:19.943375 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:19.943386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:19.943406 (+ 20us) server_negotiation.cc:407] Connection header received
0806 11:46:19.943456 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:19.943461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:19.943654 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:19.943795 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:19.944225 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.945462 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.946404 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.946775 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.946882 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:19.950129 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:19.950143 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:19.950160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:19.950191 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:19.953738 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:19.954373 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:19.954382 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:19.954384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:19.954486 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:19.954907 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:19.954913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:19.954915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:19.955911 (+ 996us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:19.957100 (+ 1189us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:19.957240 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 11:46:19.957415 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"spinlock_wait_cycles":19328,"thread_start_us":155,"threads_started":1}
I20260806 11:46:19.957945 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.944406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.0.113.61:40225)
0806 11:46:19.944746 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:19.944753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:19.944769 (+ 16us) server_negotiation.cc:407] Connection header received
0806 11:46:19.944792 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:19.944794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:19.944822 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:19.944911 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:19.945264 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.945937 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.947133 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:19.947427 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:19.947529 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:19.950900 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:19.950904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:19.950906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:19.950929 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:19.954624 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:19.955073 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:19.955078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:19.955079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:19.955681 (+ 602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:19.956013 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:19.956017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:19.956020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:19.956254 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:19.956482 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:19.957445 (+ 963us) server_negotiation.cc:299] Negotiation successful
0806 11:46:19.957599 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":105,"threads_started":1}
I20260806 11:46:19.960836 1206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:19.960937 1312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:19.961783 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.961407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38318)
0806 11:46:19.961576 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.961682 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 11:46:19.962096 1244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
I20260806 11:46:19.962260 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.961993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38322)
0806 11:46:19.962143 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.962196 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 11:46:19.963129 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.962798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38348)
0806 11:46:19.962982 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.963042 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260806 11:46:19.963158 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.962779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38332)
0806 11:46:19.963005 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.963072 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260806 11:46:19.963312 1206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:45599: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
W20260806 11:46:19.963559 1312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:45599: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
I20260806 11:46:19.994467 1206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } attempt: 1
I20260806 11:46:19.995424 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.995126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38360)
0806 11:46:19.995268 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.995337 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 11:46:19.996129 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:19.995904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38362)
0806 11:46:19.996020 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:19.996069 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 11:46:19.996259 1206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:45599: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
I20260806 11:46:20.012863 1312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } attempt: 1
I20260806 11:46:20.013824 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.013472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38366)
0806 11:46:20.013650 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.013728 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 11:46:20.014586 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.014359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38368)
0806 11:46:20.014468 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.014521 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 11:46:20.014717 1312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:45599: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
I20260806 11:46:20.071457 1206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } attempt: 2
I20260806 11:46:20.072340 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.072008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38378)
0806 11:46:20.072152 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.072222 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 11:46:20.073081 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.072876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38388)
0806 11:46:20.072978 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.073024 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 11:46:20.073237 1206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:45599: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
W20260806 11:46:20.074630 1321 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 11:46:20.074959 1321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 11:46:20.075008 1321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 11:46:20.075047 1321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 11:46:20.079123 1321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 11:46:20.079185 1321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 11:46:20.079203 1321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 11:46:20.079237 1321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 11:46:20.079280 1321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 11:46:20.082891 1312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } attempt: 2
I20260806 11:46:20.083738 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.083413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38402)
0806 11:46:20.083579 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.083648 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 11:46:20.084501 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.084262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38414)
0806 11:46:20.084382 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.084435 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 11:46:20.084633 1312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.113.60:45599: Network error: Client connection negotiation failed: client connection to 127.0.113.60:45599: connect: Connection refused (error 111)
I20260806 11:46:20.085489 1321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.113.20:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.113.62:43627,127.0.113.61:40225,127.0.113.60:45599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/krb5kdc/kudu_127.0.113.60.keytab
--principal=kudu/127.0.113.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.113.60:45599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.113.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 10:32:43 UTC on ccycloud.quasar-kjnemz.root.comops.site
build id 4570061
I20260806 11:46:20.086598 1321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 11:46:20.087666 1321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 11:46:20.097258 1330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:20.097273 1329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 11:46:20.097273 1333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 11:46:20.097579 1321 server_base.cc:1034] running on GCE node
I20260806 11:46:20.098212 1321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 11:46:20.099442 1321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 11:46:20.100605 1321 hybrid_clock.cc:648] HybridClock initialized: now 1786016780100588 us; error 31 us; skew 500 ppm
Aug 06 11:46:20 dist-test-slave-45dx krb5kdc[1084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 11:46:20.103488 1321 init.cc:377] Logged in from keytab as kudu/127.0.113.60@KRBTEST.COM (short username kudu)
I20260806 11:46:20.104467 1321 webserver.cc:460] Webserver started at http://127.0.113.60:34997/ using document root <none> and password file <none>
I20260806 11:46:20.105121 1321 fs_manager.cc:359] Metadata directory not provided
I20260806 11:46:20.105211 1321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 11:46:20.105506 1321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 11:46:20.107733 1321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/data/instance:
uuid: "2fc66ca5272d4263b8da04018f63f05f"
format_stamp: "Formatted at 2026-08-06 11:46:20 on dist-test-slave-45dx"
server_key: "aeb7ecfa31cc57eac7e0816f954b2651"
server_key_iv: "babf6caaba491d3ae22004ec9e740bbe"
server_key_version: "encryptionkey@0"
I20260806 11:46:20.108537 1321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/wal/instance:
uuid: "2fc66ca5272d4263b8da04018f63f05f"
format_stamp: "Formatted at 2026-08-06 11:46:20 on dist-test-slave-45dx"
server_key: "aeb7ecfa31cc57eac7e0816f954b2651"
server_key_iv: "babf6caaba491d3ae22004ec9e740bbe"
server_key_version: "encryptionkey@0"
I20260806 11:46:20.112352 1321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 11:46:20.114897 1343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 11:46:20.116237 1321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 11:46:20.116456 1321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/data,/tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/wal
uuid: "2fc66ca5272d4263b8da04018f63f05f"
format_stamp: "Formatted at 2026-08-06 11:46:20 on dist-test-slave-45dx"
server_key: "aeb7ecfa31cc57eac7e0816f954b2651"
server_key_iv: "babf6caaba491d3ae22004ec9e740bbe"
server_key_version: "encryptionkey@0"
I20260806 11:46:20.116597 1321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 11:46:20.130138 1321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 11:46:20.133731 1321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 11:46:20.133971 1321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 11:46:20.142288 1321 rpc_server.cc:307] RPC server started. Bound to: 127.0.113.60:45599
I20260806 11:46:20.142604 1423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.113.60:45599 every 8 connection(s)
I20260806 11:46:20.143380 1321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/data/info.pb
I20260806 11:46:20.146032 452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu as pid 1321
I20260806 11:46:20.146207 452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg6qW6_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786016488822243-452-0/minicluster-data/master-2/wal/instance
I20260806 11:46:20.146528 452 external_mini_cluster.cc:1428] Setting key 849dc6d01be67dc0edcaab45bf610c7b
I20260806 11:46:20.146549 1425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 11:46:20.152123 1425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 11:46:20 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
Aug 06 11:46:20 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:20.165376 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.148046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.0.113.60:45599)
0806 11:46:20.148524 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.148544 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.148590 (+ 46us) server_negotiation.cc:407] Connection header received
0806 11:46:20.149623 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.149651 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.150127 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.150525 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.151243 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.152779 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.154215 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.155068 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.155262 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.158672 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.158678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.158689 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.158721 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.161151 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.161794 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.161801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.161808 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.161978 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.162336 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.162342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.162346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.163043 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.163267 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.163654 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.164046 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"spinlock_wait_cycles":9600,"thread_start_us":116,"threads_started":1}
I20260806 11:46:20.166323 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.154507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.0.113.62:43627)
0806 11:46:20.154647 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.154652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.155733 (+ 1081us) server_negotiation.cc:407] Connection header received
0806 11:46:20.155861 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.155864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.155923 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.156011 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.157205 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.157894 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.158991 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.159220 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.159339 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.162193 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.162196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.162198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.162225 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.164181 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.164752 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.164756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.164758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.164863 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.165215 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.165218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.165220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.165502 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.165604 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.166054 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.166171 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 11:46:20.166352 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.154442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.62:43627 (local address 127.0.0.1:38168)
0806 11:46:20.155563 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.155626 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:20.155763 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:20.156209 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:20.156215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:20.156798 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:20.157032 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.157042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.158117 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.158120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.158850 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.158861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.159344 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.159348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.159473 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.159991 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:20.160003 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:20.162002 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:20.164380 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.164394 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.164398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.164623 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.164996 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.164999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.165000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.165099 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.165661 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:20.165667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:20.165793 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 11:46:20.165959 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 11:46:20.166162 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"spinlock_wait_cycles":2816,"thread_start_us":126,"threads_started":1}
I20260806 11:46:20.167585 1425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:20.172391 1206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } attempt: 3
Aug 06 11:46:20 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
Aug 06 11:46:20 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.62@KRBTEST.COM
I20260806 11:46:20.175793 1312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } attempt: 3
I20260806 11:46:20.177754 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.168268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.0.113.61:40225)
0806 11:46:20.168425 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.168429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.168590 (+ 161us) server_negotiation.cc:407] Connection header received
0806 11:46:20.168826 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.168830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.168900 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.169021 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.169445 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.170277 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.171089 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.171339 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.171459 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.173691 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.173694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.173697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.173744 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.175822 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.176360 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.176364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.176366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.176472 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.176804 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.176808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.176810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.177009 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.177127 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.177500 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.177624 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 11:46:20.177753 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.168155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.61:40225 (local address 127.0.0.1:55824)
0806 11:46:20.168503 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.168514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:20.168636 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:20.169036 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:20.169039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:20.169221 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:20.169315 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.169321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.170425 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.170428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.170965 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.170974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.171470 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.171473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.171578 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.171978 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:20.171989 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:20.173534 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:20.175952 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.175956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.175958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.176217 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.176603 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.176605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.176607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.176699 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.177171 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:20.177174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:20.177277 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 11:46:20.177418 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 11:46:20.177575 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288}
Aug 06 11:46:20 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.62@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
I20260806 11:46:20.179252 1425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } has no permanent_uuid. Determining permanent_uuid...
I20260806 11:46:20.179716 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.168247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.0.113.62:43627)
0806 11:46:20.168395 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.168401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.168438 (+ 37us) server_negotiation.cc:407] Connection header received
0806 11:46:20.168597 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.168600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.168652 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.168747 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.169390 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.169905 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.170730 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.170961 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.171075 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.174307 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.174311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.174314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.174350 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.176980 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.177609 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.177613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.177617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.177728 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.178456 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.178462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.178467 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.178709 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.178976 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.179400 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.179540 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Aug 06 11:46:20 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.61@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
I20260806 11:46:20.185832 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.172868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38430)
0806 11:46:20.173020 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.173036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:20.173117 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:20.174212 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:20.174215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:20.174383 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:20.174473 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.174478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.175502 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.175508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.176405 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.176421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.177106 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.177110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.177199 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.177947 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:20.177964 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:20.180690 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:20.183812 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.183817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.183819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.184092 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.184498 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.184505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.184507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.184608 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.185157 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:20.185161 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:20.185300 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 11:46:20.185429 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 11:46:20.185605 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
Aug 06 11:46:20 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.113.60@KRBTEST.COM for kudu/127.0.113.60@KRBTEST.COM
I20260806 11:46:20.186576 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.173829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.0.113.60:45599)
0806 11:46:20.173987 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.173993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.174011 (+ 18us) server_negotiation.cc:407] Connection header received
0806 11:46:20.174050 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.174053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.174101 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.174192 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.174591 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.175367 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.176567 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.176974 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.177087 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.180841 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.180845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.180848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.180875 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.183657 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.184225 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.184229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.184230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.184375 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.184732 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.184737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.184742 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.185029 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.185159 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.186219 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.186410 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":54}
I20260806 11:46:20.189666 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.176232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38434)
0806 11:46:20.176542 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.176558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:20.176634 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:20.178553 (+ 1919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:20.178560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:20.178758 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:20.178892 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.178900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.180479 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.180485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.181225 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.181236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.181930 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.181934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.182024 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.182573 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:20.182588 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:20.184435 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:20.187694 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.187698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.187700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.187943 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.188432 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.188435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.188437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.188542 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.189064 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:20.189068 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:20.189199 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 11:46:20.189302 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 11:46:20.189476 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
Aug 06 11:46:20 dist-test-slave-45dx krb5kdc[1084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786016779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.113.61@KRBTEST.COM
I20260806 11:46:20.191123 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.176354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.0.113.60:45599)
0806 11:46:20.178282 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.178289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.178301 (+ 12us) server_negotiation.cc:407] Connection header received
0806 11:46:20.178355 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.178359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.178411 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.178508 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.179037 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.180340 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.181402 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.181761 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.182256 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.184582 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.184586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.184589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.184613 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.187602 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.188145 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.188149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.188151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.188268 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.188648 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.188653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.188656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.188926 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.190622 (+ 1696us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.190708 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.190861 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"spinlock_wait_cycles":2304,"thread_start_us":177,"threads_started":1}
I20260806 11:46:20.191370 1206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124: Bootstrap starting.
I20260806 11:46:20.192416 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.179865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.113.60:45599 (local address 127.0.0.1:38450)
0806 11:46:20.180876 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0806 11:46:20.180890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 11:46:20.180965 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 11:46:20.181505 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 11:46:20.181507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 11:46:20.181688 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 11:46:20.181820 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.181827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.183097 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.183103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.183887 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 11:46:20.183898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.184598 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.184601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 11:46:20.184674 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.185200 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 11:46:20.185213 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 11:46:20.186887 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 11:46:20.190109 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.190112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.190114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.190342 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.190775 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 11:46:20.190778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 11:46:20.190780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 11:46:20.191156 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 11:46:20.191884 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 11:46:20.191889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 11:46:20.191982 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 11:46:20.192082 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 11:46:20.192225 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931}
I20260806 11:46:20.193034 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.180083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.0.113.60:45599)
0806 11:46:20.180477 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.180486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.181031 (+ 545us) server_negotiation.cc:407] Connection header received
0806 11:46:20.181055 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.181057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.181094 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.181499 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.181944 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.182962 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.184004 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.184405 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.184512 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.187041 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.187045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.187048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.187074 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.189990 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.190567 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.190573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.190578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.190682 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.191276 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.191282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.191284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.191754 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.191847 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.192746 (+ 899us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.192867 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":311,"spinlock_wait_cycles":7808,"thread_start_us":139,"threads_started":1}
I20260806 11:46:20.194585 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.181972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.0.113.61:40225)
0806 11:46:20.182073 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.182078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.182092 (+ 14us) server_negotiation.cc:407] Connection header received
0806 11:46:20.182127 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.182130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.182176 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.182256 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.183644 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.184408 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.186339 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.186667 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.186761 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.190539 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.190545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.190551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.190583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.192259 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.192886 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.192890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.192892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.192987 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.193357 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.193361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.193363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.193606 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.193798 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.194189 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.194444 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 11:46:20.195066 1312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc: Bootstrap starting.
I20260806 11:46:20.195360 1206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:20.195966 1425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f: Bootstrap starting.
I20260806 11:46:20.196067 1206 log.cc:824] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:20.198518 1206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124: No bootstrap required, opened a new log
I20260806 11:46:20.198825 1425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:20.199237 1312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc: Neither blocks nor log segments found. Creating new log.
I20260806 11:46:20.199782 1425 log.cc:824] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:20.200245 1312 log.cc:824] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc: Log is configured to *not* fsync() on all Append() calls
I20260806 11:46:20.200623 452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 11:46:20.202032 1167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5e5a2c25f5bb4c659f329b4886998124"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38176
I20260806 11:46:20.202620 1312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc: No bootstrap required, opened a new log
I20260806 11:46:20.202744 1206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } }
I20260806 11:46:20.202955 1206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a2c25f5bb4c659f329b4886998124, State: Initialized, Role: FOLLOWER
I20260806 11:46:20.203281 1206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } }
I20260806 11:46:20.203781 1437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.203938 1437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:20.204707 1206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:20.205894 1425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f: No bootstrap required, opened a new log
I20260806 11:46:20.206794 1312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } }
I20260806 11:46:20.207017 1312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df44e94e0a74bde9fc4f73549d9c0fc, State: Initialized, Role: FOLLOWER
I20260806 11:46:20.207338 1312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } }
I20260806 11:46:20.208551 1167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 11:46:20.208703 1167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:20.209388 1312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:20.210883 1167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } }
W20260806 11:46:20.211319 1455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:20.211390 1455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:20.209702 1104 master_runner.cc:186] Error getting master registration for 127.0.113.61:40225: OK, instance_id { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" instance_seqno: 1786016779915240 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 11:46:20.212880 1167 leader_election.cc:288] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [CANDIDATE]: Term 1 election: Requested vote from peers 3df44e94e0a74bde9fc4f73549d9c0fc (127.0.113.61:40225), 2fc66ca5272d4263b8da04018f63f05f (127.0.113.60:45599)
I20260806 11:46:20.212961 1104 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 11:46:20.215112 1470 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:20.215199 1470 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:20.214794 1457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.212610 1278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e5a2c25f5bb4c659f329b4886998124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc"
I20260806 11:46:20.216004 1278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:20.216089 1425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } }
I20260806 11:46:20.216382 1425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc66ca5272d4263b8da04018f63f05f, State: Initialized, Role: FOLLOWER
I20260806 11:46:20.216753 1425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } }
I20260806 11:46:20.217516 1278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a2c25f5bb4c659f329b4886998124 in term 1.
I20260806 11:46:20.217969 1127 leader_election.cc:302] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3df44e94e0a74bde9fc4f73549d9c0fc, 5e5a2c25f5bb4c659f329b4886998124; no voters:
I20260806 11:46:20.218168 1425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 11:46:20.218276 1437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 11:46:20.218519 1437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5a2c25f5bb4c659f329b4886998124, State: Running, Role: LEADER
I20260806 11:46:20.218858 1437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } }
I20260806 11:46:20.219275 1457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:20.220429 1380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e5a2c25f5bb4c659f329b4886998124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fc66ca5272d4263b8da04018f63f05f"
I20260806 11:46:20.220719 1380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [term 0 FOLLOWER]: Advancing to term 1
I20260806 11:46:20.222177 1380 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a2c25f5bb4c659f329b4886998124 in term 1.
I20260806 11:46:20.223151 1478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.224920 1478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [sys.catalog]: This master's current role is: FOLLOWER
W20260806 11:46:20.226517 1499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 11:46:20.226594 1499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 11:46:20.227828 1479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e5a2c25f5bb4c659f329b4886998124. Latest consensus state: current_term: 1 leader_uuid: "5e5a2c25f5bb4c659f329b4886998124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.228215 1479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:20.228735 1217 master_runner.cc:263] No leader master found from master 3df44e94e0a74bde9fc4f73549d9c0fc
I20260806 11:46:20.228806 1217 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 11:46:20.232672 1505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 11:46:20.234095 1505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 11:46:20.240957 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.219073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.0.113.60:45599)
0806 11:46:20.219217 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.219222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.219795 (+ 573us) server_negotiation.cc:407] Connection header received
0806 11:46:20.220460 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.220464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.220523 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.220581 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.228958 (+ 8377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.229808 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.231774 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.232169 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.232255 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.236449 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.236456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.236463 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.236499 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.238538 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.239094 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.239098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.239100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.239195 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.239542 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.239546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.239548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.239803 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.240053 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.240615 (+ 562us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.240768 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 11:46:20.249722 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.219701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.0.113.62:43627)
0806 11:46:20.219815 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.219820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.239663 (+ 19843us) server_negotiation.cc:407] Connection header received
0806 11:46:20.239859 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.239862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.239916 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.239985 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.240898 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.241679 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.243793 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.244144 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.244223 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.246142 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.246146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.246149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.246177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.247809 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.248332 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.248336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.248338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.248426 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.248744 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.248748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.248750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.248965 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.249077 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.249401 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.249541 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 11:46:20.251889 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:20.220442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.0.113.61:40225)
0806 11:46:20.220554 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.220560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.241902 (+ 21342us) server_negotiation.cc:407] Connection header received
0806 11:46:20.242098 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.242102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.242155 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.242209 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.243116 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.243888 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.245040 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:20.245401 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:20.245477 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:20.247690 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 11:46:20.247694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 11:46:20.247697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 11:46:20.247726 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 11:46:20.249710 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.250507 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.250511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.250513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.250605 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 11:46:20.250898 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 11:46:20.250901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 11:46:20.250903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 11:46:20.251117 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 11:46:20.251233 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:20.251596 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 11:46:20.251716 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 11:46:20.260509 1380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [term 1 FOLLOWER]: Refusing update from remote peer 5e5a2c25f5bb4c659f329b4886998124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:46:20.260716 1278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [term 1 FOLLOWER]: Refusing update from remote peer 5e5a2c25f5bb4c659f329b4886998124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 11:46:20.261269 1479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:46:20.265661 1479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 11:46:20.268213 1478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e5a2c25f5bb4c659f329b4886998124. Latest consensus state: current_term: 1 leader_uuid: "5e5a2c25f5bb4c659f329b4886998124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.268374 1478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:20.271867 1457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e5a2c25f5bb4c659f329b4886998124. Latest consensus state: current_term: 1 leader_uuid: "5e5a2c25f5bb4c659f329b4886998124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.272019 1457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:20.279749 1479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e5a2c25f5bb4c659f329b4886998124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.279927 1479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:20.281581 1437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e5a2c25f5bb4c659f329b4886998124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.281730 1437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124 [sys.catalog]: This master's current role is: LEADER
I20260806 11:46:20.282910 1457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e5a2c25f5bb4c659f329b4886998124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.282910 1478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e5a2c25f5bb4c659f329b4886998124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a2c25f5bb4c659f329b4886998124" member_type: VOTER last_known_addr { host: "127.0.113.62" port: 43627 } } peers { permanent_uuid: "3df44e94e0a74bde9fc4f73549d9c0fc" member_type: VOTER last_known_addr { host: "127.0.113.61" port: 40225 } } peers { permanent_uuid: "2fc66ca5272d4263b8da04018f63f05f" member_type: VOTER last_known_addr { host: "127.0.113.60" port: 45599 } } }
I20260806 11:46:20.283041 1457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:20.283042 1478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 11:46:20.283402 1508 mvcc.cc:205] Tried to move back new op lower bound from 7315524731940012032 to 7315524731780313088. Current Snapshot: MvccSnapshot[applied={T|T < 7315524731940012032}]
I20260806 11:46:20.283591 1507 mvcc.cc:205] Tried to move back new op lower bound from 7315524731940012032 to 7315524731780313088. Current Snapshot: MvccSnapshot[applied={T|T < 7315524731940012032}]
I20260806 11:46:20.283643 1513 mvcc.cc:205] Tried to move back new op lower bound from 7315524731940012032 to 7315524731780313088. Current Snapshot: MvccSnapshot[applied={T|T < 7315524731940012032}]
I20260806 11:46:20.289170 1505 catalog_manager.cc:1347] Generated new cluster ID: 094c9caef51948f09c635db4b4707d16
I20260806 11:46:20.289268 1505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 11:46:20.320432 1505 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 11:46:20.322070 1505 catalog_manager.cc:1504] Loading token signing keys...
I20260806 11:46:20.357460 1505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e5a2c25f5bb4c659f329b4886998124: Generated new TSK 0
I20260806 11:46:20.358506 1505 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 11:46:20.399499 1431 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.395833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.0.113.60:45599)
0806 11:46:20.396009 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.396014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.398105 (+ 2091us) server_negotiation.cc:407] Connection header received
0806 11:46:20.398347 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.398351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.398410 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.398466 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.399332 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38478: BlockingRecv error: recv got EOF from 127.0.0.1:38478 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 11:46:20.401093 1213 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.396650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.0.113.62:43627)
0806 11:46:20.396763 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.396767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.399952 (+ 3185us) server_negotiation.cc:407] Connection header received
0806 11:46:20.400138 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.400141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.400198 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.400248 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.400951 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38202: BlockingRecv error: recv got EOF from 127.0.0.1:38202 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 11:46:20.405998 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.397400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.0.113.61:40225)
0806 11:46:20.397546 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.397550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.401611 (+ 4061us) server_negotiation.cc:407] Connection header received
0806 11:46:20.404677 (+ 3066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.404682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.404743 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.404801 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.405644 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55850: BlockingRecv error: recv got EOF from 127.0.0.1:55850 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 11:46:20.412591 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.410627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.0.113.61:40225)
0806 11:46:20.410752 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.410757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.411380 (+ 623us) server_negotiation.cc:407] Connection header received
0806 11:46:20.411651 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.411655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.411756 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.411826 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.412465 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55856: BlockingRecv error: recv got EOF from 127.0.0.1:55856 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 11:46:20.418412 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.416534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.0.113.61:40225)
0806 11:46:20.416652 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.416657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.417339 (+ 682us) server_negotiation.cc:407] Connection header received
0806 11:46:20.417568 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.417572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.417630 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.417690 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.418293 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55858: BlockingRecv error: recv got EOF from 127.0.0.1:55858 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 11:46:20.430097 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.423019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.0.113.61:40225)
0806 11:46:20.423142 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.423147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.428775 (+ 5628us) server_negotiation.cc:407] Connection header received
0806 11:46:20.429082 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.429086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.429153 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.429217 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.429954 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55874: BlockingRecv error: recv got EOF from 127.0.0.1:55874 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 11:46:20.438791 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.437054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.0.113.61:40225)
0806 11:46:20.437208 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.437212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.437814 (+ 602us) server_negotiation.cc:407] Connection header received
0806 11:46:20.438022 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.438025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.438079 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.438130 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.438668 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55878: BlockingRecv error: recv got EOF from 127.0.0.1:55878 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 11:46:20.451932 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.449909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.0.113.61:40225)
0806 11:46:20.450039 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.450044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.450636 (+ 592us) server_negotiation.cc:407] Connection header received
0806 11:46:20.451034 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.451038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.451134 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.451192 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.451786 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55884: BlockingRecv error: recv got EOF from 127.0.0.1:55884 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 11:46:20.472151 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.470761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.0.113.61:40225)
0806 11:46:20.470927 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.470931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.471204 (+ 273us) server_negotiation.cc:407] Connection header received
0806 11:46:20.471309 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.471311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.471367 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.471446 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.472004 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55896: BlockingRecv error: recv got EOF from 127.0.0.1:55896 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 11:46:20.508492 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.507113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.0.113.61:40225)
0806 11:46:20.507301 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.507306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.507461 (+ 155us) server_negotiation.cc:407] Connection header received
0806 11:46:20.507653 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.507659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.507717 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.507812 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.508353 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55900: BlockingRecv error: recv got EOF from 127.0.0.1:55900 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260806 11:46:20.576287 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.574904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.0.113.61:40225)
0806 11:46:20.575067 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.575072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.575280 (+ 208us) server_negotiation.cc:407] Connection header received
0806 11:46:20.575475 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.575478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.575532 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.575632 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.576147 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55908: BlockingRecv error: recv got EOF from 127.0.0.1:55908 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 11:46:20.707962 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.706663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.0.113.61:40225)
0806 11:46:20.706832 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.706838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.707078 (+ 240us) server_negotiation.cc:407] Connection header received
0806 11:46:20.707188 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.707191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.707253 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.707358 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.707837 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55922: BlockingRecv error: recv got EOF from 127.0.0.1:55922 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 11:46:20.967775 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 11:46:20.966481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.0.113.61:40225)
0806 11:46:20.966660 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:20.966667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:20.966761 (+ 94us) server_negotiation.cc:407] Connection header received
0806 11:46:20.966962 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:20.966966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:20.967024 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:20.967123 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 11:46:20.967645 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55926: BlockingRecv error: recv got EOF from 127.0.0.1:55926 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 11:46:21.216724 1470 catalog_manager.cc:1259] Loaded cluster ID: 094c9caef51948f09c635db4b4707d16
I20260806 11:46:21.216823 1470 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc: loading cluster ID for follower catalog manager: success
I20260806 11:46:21.218529 1470 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc: acquiring CA information for follower catalog manager: success
I20260806 11:46:21.219903 1470 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3df44e94e0a74bde9fc4f73549d9c0fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 11:46:21.227813 1499 catalog_manager.cc:1259] Loaded cluster ID: 094c9caef51948f09c635db4b4707d16
I20260806 11:46:21.227885 1499 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f: loading cluster ID for follower catalog manager: success
I20260806 11:46:21.229550 1499 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f: acquiring CA information for follower catalog manager: success
I20260806 11:46:21.230710 1499 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2fc66ca5272d4263b8da04018f63f05f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 11:46:21.486079 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:21.482209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.0.113.61:40225)
0806 11:46:21.482494 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:21.482498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:21.482515 (+ 17us) server_negotiation.cc:407] Connection header received
0806 11:46:21.482618 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:21.482621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:21.482756 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:21.482856 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:21.483116 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.483634 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.484576 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.484799 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.484880 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:21.485208 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:21.485646 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:21.485740 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:21.485800 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 11:46:21.485851 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260806 11:46:21.492686 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:21.489179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.0.113.60:45599)
0806 11:46:21.489432 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:21.489436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:21.489451 (+ 15us) server_negotiation.cc:407] Connection header received
0806 11:46:21.489535 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:21.489537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:21.489651 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:21.489755 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:21.489936 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.490455 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.491224 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.491445 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.491522 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:21.491780 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:21.492217 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:21.492373 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:21.492424 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 11:46:21.492481 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260806 11:46:21.500003 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:21.495780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.0.113.61:40225)
0806 11:46:21.495968 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:21.495974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:21.496349 (+ 375us) server_negotiation.cc:407] Connection header received
0806 11:46:21.496546 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:21.496550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:21.496608 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:21.496684 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:21.496942 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.497532 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.498792 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.499087 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.499201 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:21.499523 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:21.499595 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:21.499690 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:21.499791 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 11:46:21.499852 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 11:46:21.500597 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:21.495794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.0.113.62:43627)
0806 11:46:21.496064 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:21.496069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:21.496400 (+ 331us) server_negotiation.cc:407] Connection header received
0806 11:46:21.496517 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:21.496522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:21.496723 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:21.496843 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:21.497112 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.497674 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.498550 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.498875 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.498985 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:21.499316 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:21.499701 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:21.499791 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:21.500351 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 11:46:21.500416 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260806 11:46:21.500921 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 11:46:21.495985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.0.113.60:45599)
0806 11:46:21.496107 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 11:46:21.496113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 11:46:21.496381 (+ 268us) server_negotiation.cc:407] Connection header received
0806 11:46:21.496608 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 11:46:21.496612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 11:46:21.496669 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 11:46:21.496736 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 11:46:21.497844 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.498620 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.499661 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 11:46:21.499998 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 11:46:21.500108 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 11:46:21.500448 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 11:46:21.500528 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 11:46:21.500574 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 11:46:21.500745 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 11:46:21.500800 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 11:46:21.503906 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 1104
I20260806 11:46:21.519908 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 1217
I20260806 11:46:21.531317 452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg6qW6_/build/debug/bin/kudu with pid 1321
2026-08-06T11:46:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3399 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6804 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (292693 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST